Method of controlling an action, such as a sharpness modification, using a colour digital image

ABSTRACT

A method for activating a function by using a measurement taken from at least one digital image having at least two colors and originating from an image-capturing apparatus, wherein a relative sharpness is measured between at least two colors on at least one region of the digital image, and at least one function is activated depending on the measured relative sharpness.

FIELD OF THE INVENTION

The invention relates to a method for activating a function, namely analteration of sharpness, using a colour digital image. It concerns moreparticularly, though not exclusively, an improvement of the sharpness ofat least one colour of a digital image. The invention also concerns asystem implementing such method, as well as an image generated by suchmethod.

The invention further concerns an embodiment for an image-capturingand/or reproducing apparatus comprising an optical system for capturingand/or reproducing images, an image sensor and/or generator and/or aservo-control system, the image being processed, in view of itsimprovement, by digital image-processing means.

The invention also concerns an apparatus obtained by such embodimentmethod.

Problem Concerned

The satisfactory visualisation of an image requires sharpness to be allthe more important than such image represents detail in compactdimensions.

Thereby, it is well known to seek to improve the sharpness of at leastone colour in a digital image according to methods, such as describedbelow:

i) In this specific case of cameras:

-   -   it is possible to use an optical focussing device (focus) which        moves the optical elements enabling to have a varied range of        distances for which the image is sharp. Such a device, manual or        motorized, often comprises a servo-control system enabling to        choose the movement depending on the distances of the objects in        the scene.

The applications of such a method are cameras and cine-cameras. It hasthe inconvenience of having a limited depth of field, especially at wideaperture, and a cost and overall size that are not easily adapted tosmall-size devices, such as telephones.

-   -   It is possible to employ a wavefront coding type solution, which        adds a specific optical element to the optical system, thus        enabling reconstruction by calculating sharpness with a larger        depth of field.

The applications of such a method are limited (microscopy) and representdisadvantages requiring a specific optical element as well as anadjustment to the hardware, including the optical system.

-   -   It is possible to implement a solution that adds a specific        optical element comprised of a flexible liquid lens that is        fixed in relation to the optics. Such a method represents a        servo-control system enabling to choose the shape of the said        lens depending on the distances of the objects in the scene.

The applications of this solution (for cameraphones or cameras) have thedisadvantage of being a specific manufacturing method, of being costlyand having a bulky optical element, and of requiring a hardwareadaptation.

ii) In a more general context, the solutions are:

-   -   deblurring algorithms on the brightness or on a colour, by        increasing sharpness via the “sharpen” method or any another        method of calculation.

The applications of such a method (all picture-taking apparatus)represent the disadvantages of having a limited increase in sharpnessand thus a very slight increase in the depth of field.

Furthermore, the known design or embodiment techniques of suchimage-capturing and reproducing apparatus, such as digital or argenticcameras, consist of first selecting the properties of the hardwareelements of the apparatus, namely the optical system, the sensor and theservo-control system. Then, as necessary, digital image-processing meansare provided for, in order to correct the defects of at least one of thehardware elements of the apparatus.

In particular, in order to design an optical system for an apparatus, itis first necessary to compile a requirements' specification charter,i.e., stating the overall dimensions, the focal length ranges, theaperture ranges, the field covered, the performances expressed, eitherin image spot size or in MTF (Modular Transfer Function), and the cost.Using such requirements' specification, the type of optical system canbe selected and, using an optical calculation software tool, such as the“Zemax” tool, the parameters of this system can be calculated, enablingto comply with the requirements' specifications as far as possible. Suchoptical system focussing is performed interactively. Generally-speaking,an optical system is designed for the purpose of representing the bestquality at the centre of the image, while the quality of the imageborders is usually of an inferior quality.

Furthermore, the common techniques are such that the optical system isdesigned in order to obtain a determined level of distortion, ofvignetting, of blur and of depth of field, thus enabling to compare theoptical system with other optical systems.

Moreover, for digital photographic apparatus, the sensor specificationsare also stated, namely: the quality of the pixels, the surface area ofthe pixels, the number of pixels, the micro-lens matrix, theanti-aliasing filters, the geometry of the pixels and the layout of thepixels.

The common technique consists of selecting the sensor of animage-capturing apparatus independently from the other parts of theapparatus and, notably, from the image-processing system.

An image-capturing and/or generating apparatus also commonly comprisesone or several servo-control systems, such as an exposure system and/ora focussing system (automatic focus or “autofocus”) and/or aflash-control system.

Thereby, in order to specify an exposure system which activates theaperture and the exposure time, possibly the sensor gain, means formeasuring are determined; shall be especially determined the image zoneson which exposure shall be measured, in addition to the weight affectedto each zone.

For a focussing system, shall be determined the number and the positionof image zones to be used for focussing. Shall also be specified, forexample, recommendations for the driver movement.

Whatever the case, such specifications are applied, regardless ofwhether there exist digital means for image-processing or not.

THE INVENTION Observations Concerning the Invention

The invention stems from the combination of the following observations,which concern it alone:

i) The image-capturing and/or processing apparatus generate on suchimages a variable sharpness, which depends upon the considered colour,as described below by way of FIGS. 1 a and 1 b.

FIG. 1 a shows the converging lens 1 of an optical device (notillustrated) equipped with a sensor 2 located on a point 3.2 of focusassociated to a wavelength λ2. Hence, the colour defined by this lengthλ2 is sharp on an image formed by such lens when the image represents anobject way off in the distance.

Nevertheless, such adjustment entails three problems:

-   -   First of all, the point 3.2 of the lens focus directly relates        to the colour defined by this wavelength λ2, such that a point        of focus 3.1 directly relating to another colour defined by a        wavelength λ1 is located upstream from the sensor.

Consequently, the image formed by this second colour (λ1) at sensorlevel is not as sharp as the image formed by the first colour (λ2), thusreducing the sharpness of the overall image formed by the sensor.

-   -   Secondly, the point of focus of the lens for a wavelength is        variable, depending on the distance separating it from object 4        represented on the image.

Hence, FIG. 1 b shows the new locations 4.1 and 4.2 of the points offocus, respectively associated to the wavelengths λ1 and λ2, when theobject represented passes from a far-off distance (FIG. 1 a) to a moreclose-up distance (FIG. 1 b).

In such latter case, it appears that the sensor is located on the pointof focus of the colour (λ1) which, beforehand, did not provide a sharpimage.

-   -   Thirdly, the point of focus of the lens for a wavelength and a        distant object is variable, depending on the position of the        object represented in the image.

ii) As shown in FIG. 2, which is an example of spectral distribution foran image according to axis 6.1, the images are generally comprised ofseveral colours, the intenseness of which (Y-axis 6.2) can be similar.In this example are represented the blue components 5.1 (wavelengthapproximately 450 nm), the green components 5.2 (wavelengthapproximately 550 nm) and the nearby red components (wavelengthapproximately 600 nm), although it is clear that the invention isapplied to an image regardless of its considered colour and wavelengthdistribution (for example, infrared or ultraviolet).

iii) The standard sharpness-improvement techniques do not take advantageof the fact that one of the colours can be sharper than the othersdepending on the distance of the object represented on the image.

iv) Furthermore, the invention stems from the observation that thestandard apparatus' design or embodiment techniques do not enable tofully take advantage of the possibilities offered by the digital meansof image-processing.

THE INVENTION

Thereby, the invention concerns, in a general manner, asharpness-improvement method for at least one colour of a digital image,comprising the steps of

-   -   selecting from among the image colours at least one colour        referred to as a sharp colour,    -   reflecting the sharpness of the sharp colour onto at least one        other improved colour, in order that the improved colour        represents increased sharpness.

In light of the invention, it is thus possible:

-   -   to increase the image's perceived sharpness,    -   to increase the depth of field of a capturing apparatus,    -   to create a macro function,    -   to control the depth of field, regardless of the exposure,    -   to measure the distance of the objects of an imaged scene using        an image,    -   to improve the exposure and/or focussing and/or flash        servo-control devices,    -   to reduce the costs of the capturing apparatus,    -   to reduce, in terms of equal performance, the size of the        capturing apparatus,    -   to read bar-codes and/or business cards and/or written text        and/or take portraits and/or landscape views, using a same lens,        having a fixed focus, like, for example, that of a cameraphone,    -   to design and/or choose a lens that gives the apparatus        increased specifications, notably in terms of aperture and of        depth of field,    -   to create image effects depending on the relative sharpness        between at least two colours and/or on the distance of objects        of the imaged scene,    -   to enable the user to digitally change the image focus of the        relative sharpness between at least two colours and/or of the        distance of objects of the imaged scene,    -   to reduce the time between the request for capture and the        actual image capture, by eliminating or simplifying the optics        system for focussing.

The invention further concerns an embodiment for a capturing apparatus,comprising an optical capturing system, a sensor and/or a servo-controlsystem, the image being processed, in view of its improvement, bydigital image-processing means;

a method wherein the user determines or selects the optical systemand/or the sensor and/or the servo-control system parameters, using thecapacity to process images via digital means, in order to minimiseembodiment costs and/or to optimise the performances of the capturingapparatus.

In an embodiment, the method further comprises the step of decomposingthe digital image into regions, the said sharp colour being selected foreach region.

In an embodiment, the said sharp colour selection consists of choosingthe sharpest colour according to a pre-determined rule.

In an embodiment, the said “sharp colour” selection is pre-determined.

In an embodiment, the said digital image stems from a capturingapparatus and the said sharp colour selection depends upon the distancebetween the capturing apparatus and at least one object of the capturedscene in order to obtain the said digital image.

In an embodiment, the said image-capturing apparatus comprises a macromode, the said sharp colour selection depending upon activation of themacro mode.

In an embodiment, the said digital image stems from a capturingapparatus, the said method further comprising the step for determiningthe distance between the capturing apparatus and at least one object ofthe captured scene using the sharpness of at least two colours in animage region of the said object.

In an embodiment, the method further comprises the step to reduce thesharpness of at least one colour within at least one image region.

In an embodiment, the method further comprises the step for determininga servo-control instruction for the said capturing apparatus using thesharpness of at least two colours, in order that focussing is achievedin fewer steps and is accelerated.

In an embodiment, the said digital image stemming from a capturingapparatus comprising a lens, the said method further comprises the stepfor selecting a lens from among a series of pre-determined lenses, thesaid lens representing specifications such that the images of an objecthaving at least two pre-determined distances represent distinct sharpcolours, thus improving the depth of field and/or reducing the cost ofthe lens.

In an embodiment, the said digital image stemming from a capturingapparatus comprising a lens, the said method further comprises the stepto design a lens by taking account of the method according to theinvention, the said lens representing specifications such that theimages of an object having at least two pre-determined distancesrepresent distinct sharp colours;

-   -   in order that the depth of field and/or the aperture and/or        every other optical specification is improved and/or the cost of        the optics is reduced.    -   in order that the mechanical focussing can be achieved using        fewer positions.

In an embodiment, the repercussion of the sharpness of the sharp colouron at least one other improved colour is embodied by using a calculationof the type CA=CN+F(CO−CN), where CA represents the improved colour, COrepresents the improved colour prior to processing, CN represents thesharp colour and F represents a filter, namely a low-pass filter.

The invention further concerns an embodiment for an image-capturingand/or reproducing apparatus (20) comprising an optical system (22, 22′)for capturing and/or reproducing images, an image sensor (24) and/orgenerator (24′) and/or a servo-control system (26), the image beingprocessed, in view of its improvement, by digital image-processing means(28, 28′),

-   -   the method being such that the user determines or selects the        optical system and/or the sensor and/or the image generator        and/or the servo-control system parameters, using the capacity        to process images via digital means, and especially for        improving the sharpness of a colour depending on the sharpness        of another colour in accordance with a method complying with one        of the previous claims,    -   in order to minimise the embodiment costs and/or to optimise the        performances of the image-capturing and/or reproducing        apparatus.

The invention also concerns an image-capturing and/or reproducingapparatus using a colour-improvement method according to one of thepreceding embodiments and/or obtained via an embodiment according to theprevious embodiment.

The invention also concerns a digital image obtained according to amethod complying with one of the preceding embodiments or using anapparatus complying with the previous embodiment.

Finally, the invention also concerns a digital image-processing deviceimplementing a method according to one of the preceding embodiments.

DEFINITIONS

Hereunder are noted the meanings of the various terms employed:

-   -   By digital image is meant an image produced under digital form.        The image may stem from an image-capturing apparatus.

The digital image may be represented by a series of digital values,hereinafter referred to as “grey level”, each digital value being linkedto sensitivity in terms of colour and to a relative geometrical positionon a surface or within a volume. Colour, within the meaning of theinvention, is referred to as a series of digital values linked to suchsame sensitivity in terms of colour.

The digital image is preferably the raw image from the sensor prior to“demosaicing” (i.e. removing the matrix). The digital image may alsohave been processed, for example demosaicing or white balancing.According to the invention, the digital image shall preferably not haveundergone sub-sampling.

-   -   If the digital image stems from an image-capturing apparatus,        such image-capturing apparatus shall comprise a sensor equipped        with sensitive elements. By sensitive element is meant a sensor        element enabling to convert a flow of energy into an electric        signal. The flow of energy can notably take the form of a        luminous flow, of X-rays, of a magnetic field, of an        electromagnetic field or of sound waves. The sensitive elements        may be, depending on the case, juxtaposed on a surface and/or        superimposed within a volume. The sensitive elements may be        placed according to a rectangular matrix, a hexagonal matrix or        any other geometry.    -   The invention is applied to sensors comprising sensitive        elements of at least two different types, each type having        sensitivity in terms of colour and each colour sensitivity        corresponding to the part of the energy flow converted into an        electric signal by the sensor's sensitive element. In the case        of a visible image sensor, the sensors are generally sensitive        to 3 colours, the digital image also having 3 colours: red 5.1,        green 5.2 and blue 5.3, as illustrated in FIG. 2, which shows        the amount of converted energy on the vertical axis 6.2 and the        length of the wave on the horizontal axis. Some sensors are        sensitive to 4 colours: red, green, emerald and blue.    -   By colour is also meant a combination, especially linear, of        signals emitted by the sensor.    -   The invention is applied using the various known definitions of        every known sharpness. For example, the sharpness of a colour        may correspond to the measurement of a value referred to as        “BXU”, which is a measurement of the blur spot surface, such as        described in the article published in the “Proceedings of IEEE,        International Conference of Image Processing, Singapore 2004”,        and entitled “Uniqueness of Blur Measure” by Jerôme BUZZI and        Frederic GUICHARD.

Simply-speaking, the blur of an optical system is measured from theimage, called “impulsive response”, from an infinitely small pointlocated within the sharpness plane. The BXU parameter is the variationof the impulsive response (i.e. its average surface). The processingcapacity may be limited to a maximum BXU value.

Diverse measuring methods for such sharpness are described in suchhandbooks and publications as, for example, the “Handbook of Image&Videoprocessing”, edited by Al Bovik and published by the Academic press,pages 415 to 430.

A parameter refers to the quality of an image, such as generallyaccepted. In an embodiment, the sharpness of a colour is achieved bycalculating a gradient. For example, the sharpness of a colour may beobtained by calculating a gradient of 9 levels of grey matter taken fromneighbouring geometrical positions within the colour considered.

The invention refers to the sharpness of at least two colours. Accordingto an embodiment, the sharpness of at least two colours is onlyconsidered in a relative manner, one in relation to the other. For suchembodiment, a gradient enables to simply calculate a relative sharpnessbetween two colours, irrespective of the image contents.

The invention refers to selecting, from among the colours, at least onecolour referred to as “sharp colour”. According to an embodiment, suchselection is possible by determining which colour out of at least two isthe sharpest. For such embodiment, a gradient enables to simplydetermine the sharpest colour from among at least two colours.

In an implementation,

-   -   An image-capturing apparatus is, for example, a disposable        camera, a digital camera, a reflex camera (digital or not), a        scanner, a fax machine, an endoscope, a cine-camera, a        camcorder, a surveillance camera, a game, a cine-camera or photo        apparatus integral with or linked to a telephone, to a personal        assistant or to a computer, a thermic camera, an ultrasound        apparatus, an MRI (Magnetic Resonance Imaging) apparatus, a        X-ray radiography apparatus.

It should be noted that the present invention refers to such types ofapparatus, if they process images comprising at least two colours.

-   -   By optical system for image-capturing is meant the optical means        enabling to reproduce the images on a sensor.    -   By image-capturing is meant the mechanical, chemical or        electronic means enabling to capture and/or record an image.    -   By servo-control system is meant the means, whether of the        mechanical, chemical, electronic or information technology type,        enabling the elements or parameters of the apparatus to comply        with instructions. It especially refers to the automatic        focussing system (autofocus), to the automatic white-balance        control, to the automatic exposure control, to the control of        the optical elements, in order, for example, to maintain a        uniform image quality, to an image-stabilising system, to an        optical and/or digital zoom factor control system, to a        saturation control system, or to a contrast-control system.    -   The digital image-processing means may adopt diverse forms        depending on their application.    -   The digital image-processing means may be integrated, partially        or wholly, into the apparatus, such as in the following        examples:    -   An image-capturing apparatus which produces altered images, for        example, a digital photo apparatus with integral        image-processing means.    -   An image-reproducing apparatus which displays or prints altered        images, for example, a video projector or a printer comprising        image-processing means.    -   A multi-function apparatus which corrects the defects of its        elements, for example, a scanner/printer/fax machine comprising        image-processing means.    -   A professional image-capturing apparatus which produces altered        images, for example, an endoscope comprising image-processing        means.

According to an embodiment:

-   -   the digital image-processing means include means for improving        the image quality by activating at least one of the parameters        of the group comprising: the geometric distortions of the        optical system, the chromatic aberrations of the optical system,        the compensation of the parallax, the depth of the field, the        vignetting of the optical system and/or of the image sensor        and/or generator, the lack of sharpness of the optical system        and/or of the image sensor and/or generator, the noise, the        moiré phenomena, and/or the contrast,    -   and/or the given or selected parameters of the optical system        are chosen from within the group comprising: the number the        system's optical elements, the type of the materials comprising        the optical elements of the optical system, the cost of the        materials for the optical system, the processing of the optical        surfaces, the assembly tolerances, the value of the parallax        according to the focal length, the aperture specifications, the        aperture mechanisms, the range of possible focal lengths, the        focussing specifications, the focussing mechanisms, the        anti-aliasing filters, the overall dimensions, the depth of the        field, the specifications linking the focal length and the        focussing, the geometric distortions, the chromatic aberrations,        the off-cantering, the vignetting, the sharpness specifications,    -   and/or the given or selected parameters of the image-capturing        and/or generating apparatus are chosen from within the group        comprising: the quality of the pixels, the surface area of the        pixels, the number of pixels, the micro-lens matrix, the        anti-aliasing filters, the geometry of the pixels, the layout of        the pixels.    -   and/or the given or selected parameters of the servo-control        system are chosen from within the group comprising: the        focussing measurement, the exposure measurement, the white        balance measurement, the focussing instructions, the aperture        instructions, the exposure instructions, the sensor-gain        instructions, the flashlight instructions.

For the servo-control system enabling automatic focussing, it isrecalled that focussing may be performed in various manners,particularly by controlling the position of mobile elements of theoptical system or by controlling the geometry of the flexible opticalelements.

-   -   The performances of a capturing apparatus are notably its cost,        its overall dimensions, the quantity of minimal light that it        may receive of emit, the image quality, namely its sharpness,        the technical specifications of the optics, the sensor and the        servo-control, as well as its depth of field.

Thereby, it should be noted that the depth of field can be defined asthe range of distances in which the object generates a sharp image, i.e.where the sharpness exceeds a given threshold for a colour, generallygreen, or even defined as the distance between the nearest object planeand the farthest object plane for which the blur spot does not exceedthe pre-determined dimensions.

As the colour green is predominant for defining the sharpness of animage, as subsequently explained, it is also common to use green todefine the depth of field.

The invention also concerns an apparatus obtained through the embodimentmethod, such as defined above.

According to other specifications of the invention, which may be usedseparately from or combined with those described above:

The invention concerns a method for activating a function using ameasurement performed on at least one digital image, having at least twocolours, originating from an image-capturing device, wherein:

-   -   the relative sharpness is measured between at least two colours        in at least one region R of the image, and    -   a function is activated depending on the measured relative        sharpness.

By region is meant a part of or the whole of the image. A regioncomprises one or several pixels, adjacent or not.

Hence, the action is notably adapted to the distance between the imagedobject and capturing apparatus, or is adapted to the relative depthbetween two imaged objects.

It is possible to measure the relative sharpness in various manners, forexample (though without such list being exhaustive):

-   -   the sharpest colour can be determined, and/or    -   at least one colour, referred to as “sharp colour”, can be        selected from among the colours, and/or    -   sharpness between the colours can be compared, and/or    -   a sharpness difference can be calculated, and/or    -   the relative sharpness can be directly calculated.

Various examples of relative sharpness measurements will be explainedhereafter, notably illustrated in FIGS. 3 a, 3 b, 4, 5, 6, 7, 8, 9 and10.

The relative sharpness and/or the measurement of relative sharpness in aregion may be expressed by a single digital value, for example,realising the average relative sharpness in the region, or by way ofseveral digital values realising the relative sharpness in various partsof the region.

According to the invention, at least one function is activated,depending on the measured relative sharpness. Such action is notably(though without such list being exhaustive):

-   -   direct or indirect processing (especially via the provision of        processing or distance-data and/or position and/or direction        parameters) of the digital image and/or of another digital        image, and/or    -   a distance and/or direction and/or position and/or size and/or        orientation and/or geometric form measurement of at least one        part of at least one object or subject of the scene, and/or    -   data directly or indirectly linked to the geometry of the imaged        scene in three dimensions, and/or    -   an object detection, notably a face and/or the main subject or        subjects, and/or    -   object recognition and/or authentication, for example, a face,        and/or    -   a position and/or movement measurement of the apparatus, and/or    -   a servo-control of the apparatus or of another device, such as a        robot, and/or    -   automatic framing of the main subject, and/or    -   one adjustment alteration of the apparatus, and/or    -   the production or the activation of a signal, and/or    -   an addition, an elimination or alteration of an object within        the digital image or another digital image, and/or    -   any other action directly or indirectly using the relative        sharpness measurement.

According to an embodiment, the action implements:

-   -   the digital image, and/or    -   another digital image, and/or    -   choice by the user of the apparatus, and/or    -   at least one specification of the capture apparatus during        picture-taking, and/or    -   other data.

In the event where the action relates to direct or indirect processing,the process may consist of one of the following actions (though withoutsuch list being exhaustive):

-   -   digitally altering focussing, and/or    -   creating image effects depending on the relative sharpness        between at least two colours and/or on the distance of objects        of the imaged scene, and/or    -   reducing the sharpness of at least one colour within at least        one region of an image, and/or    -   increasing the sharpness of at least one colour within at least        one region of an image, and/or    -   activating compression, and/or    -   embodying any other process described herein.

The use of the measured relative sharpness for activating the functionespecially enables the function to be adapted to the distance between atleast one part of an imaged object and the measuring apparatus, and/orto the geometry of at least one part of an object, and/or to theposition and/or the size of at least one part of the object, and/or tothe direction of at least one part of the object.

The known methods do not enable activation of such type of function asfrom a relative sharpness measurement of at least one image region, butrather require the use of a particular device, in addition to theimage-capturing apparatus, for the purpose of estimating a distance.Furthermore, the known methods only enable a distance measurement in oneparticular point or in a limited number of points, whereas the inventionenables to measure the distance in a vast number of pointssimultaneously.

According to an embodiment, the function activated is included in thegroup comprising:

-   -   determination of the distance between the capturing apparatus        and at least one object imaged by the digital image, and/or        determination of the relative distance between two imaged        objects,    -   an action depending on the said distance and/or the said        relative distance,    -   a process on at least one zone Z′ of the digital image and/or of        another digital image,    -   a servo-control of the capturing apparatus and/or a        servo-control of another apparatus,    -   the provision of an indication and/or alarm and/or alert signal        to a user,    -   the detection of a part of the image,    -   an alteration of a colour sharpness,    -   a determination of the position and/or of the movement of the        capturing apparatus,    -   the determination of a subject's position within the image,    -   an alteration of at least one image specification,    -   an alteration of all or part of the image,    -   the determination of a zone of interest inside the image,        notably in order to provide a servo-control signal,    -   the alteration of the resolution for all or part of the image,    -   the provision of data relating to the image,    -   the provision of data to a sound-capturing device,    -   the parametering of a compression,    -   an alteration of all or part of the image,    -   at least one adjustment of the capturing apparatus.

According to an embodiment, the function activated comprises a processon at least one zone Z′ of the digital image and/or of another digitalimage.

The zone Z′ is a part or not of the digital image on which the relativesharpness has been measured.

As a processing example performed on a digital image, separate from thaton which has been measured the relative sharpness between at least twocolours, can first be quoted the taking of a video sequence wherein thenext image, or another image, can be processed, such process consistingof increasing the sharpness (also given as an example).

Indeed, the sharpness of a next image can be increased, since it isbased upon the measurement of a preceding image which is hardlydistinguishable from such next image. Hence, it not necessary to savethe current digital image in the memory.

In another example: the sharpness measurement is, within a digital photoapparatus, performed on the image displayed prior to the actual takingof the picture; the image taken is processed at a later stage at fullresolution (while the measurement taken on the image displayed prior tothe actual picture-taking is generally at a lower resolution) using thelast measurement or a combination of last measurements.

In an embodiment, the zone Z′ constitutes all or part of the digitalimage region (on which the relative sharpness measurement has beentaken), and/or the whole digital image, and/or a separate zone from thedigital image region, and/or another digital image, and/or another wholedigital image.

When the zone Z′ constitutes all or part of the digital image region,for example when the depth of field needs to be increased, the zone Z′is a pixel; a region of N pixels, on which is measured the relativesharpness, is defined in accordance with such relative sharpness,wherein a filter is applied for the purpose of transporting thesharpness of the sharpest colour to the other colour in order that thesharpness of the pixel is increased. By repeating this operation foreach pixel, the depth of field is thus increased.

The zone Z′ on which is performed the process may constitute a fulldigital image, notably when the sharpness on the full image isincreased.

As an example of a process on a distinct zone of the digital imageregion, the case where the relative sharpness measurement is performedon a region shall be quoted, whereby the process is applied on a centredimage part corresponding to a digital zoom.

As an example of a process applied to a zone of another digital imageand/or to another full digital image, the above example of a videosequence is recalled, the other digital image being, for example, animage following a video image; the other image is also, for example, thedigital image taken at full resolution for a photo apparatus, whereasthe image on which the measurement is taken is at low resolution.

In an embodiment, the zone Z′, for which a process is activated,includes at least one pixel of an image, while the region includes apredetermined vicinity of the corresponding pixel in the digital image.The processed image may be the digital image. The processed image mayalso be another image, for example an image stemming from the samecapturing apparatus and captured after the digital image. In such acase, the correspondence between the pixels of the two images can beachieved by associating the pixels of the two images situated in thesame place. Such case has the advantage of preventing digital-imagestorage between the measurement and the processing without anytroublesome artifact, if the images are captured within a short timeframe, for example 1/15 s.

In an embodiment, this treatment is applied to all the pixels of animage. The processed image may be the digital image. The processed imagemay also be another image, for example an image stemming from the samecapturing apparatus and captured after the digital image.

In an embodiment, the process on at least the zone Z′ includes thealteration of at least one image specification included within the groupcomprising: sharpness, contrast, luminosity, detail, colour, the type ofcompression, the rate of compression, the image contents, theresolution.

Example of Contrast Alteration:

The contrast of the close-up objects is increased and the contrast ofthe background objects is reduced, for example in the case of avideo-conference. Conversely, the contrast of the close-up objects canbe reduced and that of the background objects can be increased in orderto diminish the blurring effect.

Example of Luminosity Alteration:

The process may involve the brightening of close-up objects and thedarkening of the background, e.g. for a video-conference. Conversely,for an image taken using the flashlight, the luminosity process shallconsist of lightening up the background and dimming the close-up objectsin order to compensate the flashlight effect.

Example of Detail Alteration:

For a video-conference, the detail of the background objects can bereduced in order to allow a higher compression for such backgroundobjects, while maintaining maximum quality for the main subject.

Example of Colour Alteration:

Saturation in terms of the colours of the regions is reduced, i.e. wherethe relative sharpness exceeds a threshold, in order to eliminate theexcessive longitudinal chromatic aberrations, sometimes called “purplefringing”.

Example of Compression-Type Alteration:

For a video-conference, for example, a near object/distant objectsegmentation is provided to a codec MPEG-4 in order to enable thedistant object to be highly compressed for the purpose of maintainingmaximum quality for the close-up main subject.

Example of Compression-Rate Alteration:

As mentioned above in the case of a video-conference, the compressionrate can be higher for the background than for the main subject.

Example of Contents Alteration:

The process consists of replacing a background by a landscape or adecor.

In an embodiment, the process includes a sharpness alteration for eachpixel of the zone Z′, by way of a filter mixing the values attached tothe pixel within a predetermined vicinity of each pixel, the parametersof the filter depending upon the measured relative sharpness.

In an embodiment the zone Z′ is determined using the measured relativesharpness.

For example, the Z′ zone corresponds to the image parts where therelative sharpness is comprised within a given range corresponding tothe parts of the image containing objects located within a given rangeof distances, which enables, for example, to separately process aforeground and a background.

Within the meaning of the invention, the French “arrière-plan” and“fond” have the same connotation of “background” as in English where nodifference is made.

In an embodiment, the zone Z′ constitutes a background for an image,notably destined for remote transmission, especially through a system ofvisio or video-conferencing. The processed image may be the digitalimage. The processed image may also be another image, for example animage stemming from the same capturing apparatus and captured after thedigital image.

According to an embodiment, the process includes the provision of datadepending on the distance between the imaged object and the capturingapparatus for all or part of the pixels of the zone Z′, and where astorage and/or a transmission and/or a use of such data is activateddepending on the distance, the stored data notably being saved in adata-processing file, namely in an image file.

It is recalled that the zone Z′ can constitute a point and/or a regionand/or several regions and/or a full image and/or a main subject and/ora background.

The data depending on the distance can be a distance with, for example,a precision indication or a range of distance values, like, for example,a distance less than one centimetre, a distance comprised between 1 and10 centimetres, then between 10 centimetres and 1 metre, and finallybeyond one metre. The data depending on the distance can also berepresented by a criterion of the “too close”, “close”, “near”, “far” or“macro” type. The data depending on the distance can also be convertedinto information on the type of objects or subjects, such as “portrait”or “landscape”.

Hence, a map of the distances of the various parts of the image may alsobe provided. It is also possible to provide the position of the zone inrelation to the capturing apparatus.

The data depending on the distance can also comprise distance values forthe various elements of the image, such as the minimum distance, themaximum distance, the average and the typical difference.

It is important to note that the invention enables to measure severaldistances within a scene using a single image, whereas the prior artrequires complex means, such as using several cameras placed in severalpositions in order to achieve the stereoscopy, or a travelling camera,or a laser range-finder, or even an ultrasound sonar which does notenable to obtain a visible image.

In an embodiment, the function activated includes a servo-controllingfunction for the capturing apparatus comprised within the groupconstituted by: a servo-control for focussing, a servo-control forexposure, a servo-control for flash, a servo-control for image-framing,a servo-control for white-balancing, a servo-control forimage-stabilising, a servo-control for another apparatus or devicelinked to the capturing apparatus, such as the guiding of a robot.

Example of a Servo-Control Focussing Function:

The main subject or the zones of interest can be detected by thedistance measurements, as from sharpness, the main subject or the zoneof interest thus being the nearest zone.

A servo-control for focussing, embodied using measurements takendirectly from a single digital image, is particularly advantageous inrelation to the known focussing servo-controls, or “autofocus”, forwhich it is necessary to take measurements from successive images.

Moreover, a known focussing servo-control consists of pressing a triggerelement until half-way down, then of moving framing before pressing downcompletely, whereas with the invention, focussing can be achieved in anentirely automatic manner; the invention thus enables a gain of time anda better image.

Example of a Servo-Control Exposure Function:

Similar to the focussing servo-control, the exposure adjustment isachieved on the main subject, which is automatically detected; henceexposure can be correct whatever the position of the main subject withinthe image frame. In other words, similar to the focussing, the user hasno need to aim at the subject, then press half-way down before movingthe framing.

Example of a Flashlight Servo-Control:

As the invention enables to determine the main subject, brighteningfunction can be activated according to the main subject, while with thestate of the art, the strength of the flashlight is adjusted inaccordance with the focussing without determination of the main subject,i.e. the nearest subject in particular. As indicated above, the subjectsin the least light can be processed digitally through brightening.

Example of Control of Another Device:

When a mobile robot has to move, the regions the nearest to the mobilerobot are determined, with a trajectory, free of all hindrance, beingdetermined as from the objects the nearest to the mobile robot.

In an embodiment, the function activated includes a provision of asignal, such as an indication signal of the main focal point of thedigital image and/or of a focussing zone, and/or an alarm signalindicating an alteration of the digitally-monitored and imaged sceneand/or of the distance of at least one part of the imaged scene, to thecapture apparatus.

For example, in a digital photo apparatus, it is possible to have aframe, notably in predetermined form, surrounding the main subject forthe purpose of informing the photographer which main subject has beendetected by the apparatus during picture-taking. Such indication signalof the main subject can notably be used prior to the actualpicture-taking in order to inform the photographer what will be thesharpest subject or object.

Such signal may also be an indication that the closest object or subjectis too close-up in relation to the picture-taking apparatus for it to besharp. In such a case, the signal takes the form, for example, of aclear message “Foreground too close”, or of an exaggeration of theforeground blur, or even of a visible alteration of the foregroundcolour.

The signal indicating that the scene or the object of the foreground istoo close-up may take account of the final destination of the image thatis to be taken, notably of the resolution selected for such destination.For example, a subject that would be blurred on a television-receiver orcomputer screen may be sharp on a small-size screen of the type found ona picture-taking apparatus. Likewise, a blurred subject for printing on24 cm×30 cm paper is not necessarily so for printing on 10 cm×15 cmpaper.

The blurred indication signal may also take account of the subject. Forexample, the detection of a bar code is more tolerant to blur than anatural image.

Example of an alarm signal provided by a picture-taking apparatus:

In a video-surveillance system monitoring an object, the picture-takingapparatus is adjusted to cover two regions. The first of these regionsis the one where the object is found, while the second region is thefull range of the picture-taking apparatus. If an object within thepicture-taking range comes closer to the object to be monitored, analarm is thus activated.

In an embodiment, the function activated depends upon at least onespecification of the capturing apparatus during picture-taking, namely,the focal length, the aperture, the focussing distance, the exposureparameters, the white-balance parameters, the resolution, thecompression, or an adjustment made by the user.

Indeed, the function activated depends upon the measured relativesharpness and such relative sharpness between at least two coloursdepends upon the adjustment of the picture-taking apparatus, namely thefocal length, the aperture and the focussing distance.

In an embodiment, the digital image constitutes a raw image stemmingfrom the sensor of the capturing apparatus.

Such function makes the relative sharpness function easier, since whenusing a raw image, the measurement is not affected by such processes asthe demosaicing, the sharpness improvement filter, the change in thecolour area or the shade curve.

The raw image stemming from the sensor may, however, have beenprocessed, for example soundproofing, digital gain, compensation of thedark level.

The relative sharpness measurement and/or the function activated may beperformed within the capturing apparatus.

The relative sharpness measurement may be performed beyond the capturingapparatus, for example on a computer after transfer of the digitalimage, and/or the user activates a function beyond the capturingapparatus.

It is indeed possible to take a relative sharpness measurement beyondthe capture apparatus; likewise the function can be activated beyond thecapturing apparatus, such as already mentioned. For example, aprocessing programme implemented on a computer determines, using thesharpness measurements, the focussing distance and/or the depth of fieldin order to implement the processes depending on such distance and/orthe depth of field.

In an embodiment, the function comprises a detection and/or recognitionfunction for a part of the image, such as face detection and/orrecognition.

For example, it is known that a face represents a given size. The methodaccording to the invention enables to determine the distance between theobjects or subjects and on the capturing apparatus. Furthermore, usingsuch distance data, for the focal length and for the size of the objectin the image, the existence of the face can be deducted (whichrepresents a size comprised within a given range). The size criterion ofthe object can be completed by other criteria, like, for example, thecolours. Detection of an object, such as the detection of faces, can beespecially used, during teleconferences, to automatically perform a highbackground compression. Such method may also be used for the detectionof a defect so as to correct it, of red eyes, or for recognising faces(biometric applications).

In an embodiment, the function activated comprises a position and/ormovement measurement of the capturing apparatus.

In an embodiment, one or several objects destined to remain fixed in ascene of captured image shall be stored in the memory, while movement orpositioning shall be detected by determining the variation of therelative sharpness over time. Such arrangement can, for example, be usedto embody a computer interface of the visual “mouse” type in threedimensions.

In an embodiment, the function activated comprises the determination ofthe position of the main subject or subjects in the image.

The determining criterion of the main subject within a digital imageshall be the shortest distance in relation to the capturing apparatus.Nevertheless, such criterion may be combined with other factors. Forexample, objects on the edge of the image that would be close to thecapturing apparatus, may be eliminated through an automatic process. Aspreviously described, it is also possible to take account of theobject's size criterion, such size depending upon the focal length andthe distance between the capturing apparatus and the object.

In an embodiment, the function activated further comprises the automaticframing, namely the centering, or the reframing of the digital imageand/or of another image on the main subject of the digital image. Thereframed image may be the digital image. The reframed image may also beanother image, for example an image stemming from the same capturingapparatus and captured after the digital image.

For example, it is possible to provide a “close-up” mode, whichautomatically ensures a framing on a foreground object. It is alsopossible to provide a “bust” mode which automatically ensures theframing of a face according to the said three-thirds' rule, for examplepositioned within a third of the image height and width.

In an embodiment, the function activated comprises the application of aprocess which, on the one hand, depends upon the relative sharpness andupon the user's selection criterion, on the other.

For example, the criterion selected is as follows: privilege the partsof the image that are the nearest to the capturing apparatus. Thereby,the function may consist of increasing the sharpness of such parts ofthe image and of reducing the sharpness of the remainder of the image inorder to create a depth of field lower than that actually achieved.Under such conditions, it is possible to simulate the behavior of a lenshaving variable focussing and aperture within an image achieved using alens without any functions, whether focussing or aperture, such as in a“cameraphone”.

In an embodiment, the function activated comprises alteration of thecontrast and/or of the brightness and/or of the colour and/or of theimage sharpness, depending on the variation of the relative sharpnesswithin the image.

Hence, it is possible to simulate localised lighting, such as that of aflashlight; it is also possible to reduce the effect of a flashlight,for example, in order to reduce the backlighting or the flat-tinteffects.

A scene is lit up by one or several natural or artificial sources, aswell as possibly by one (or several) flashlight (or lights) controlledby the apparatus.

It is known that an image-capturing apparatus controls exposure(exposure time, sensor gain and, where necessary, aperture), controlswhite balance (gain for each colour within the whole image) and possiblycontrols the flashlight (duration and strength of the flashlight),depending on the measurements in a digital image of the scene (forexample, analysis of the saturated zones, analysis of the histogram,analysis of the average colour) and/or controls the measurements takenwith a supplementary device: infrared range-finder, pre-flash for theflashlight, etc., focussing servo-control enabling to find the focusproduced by the sharpest image by comparing the sharpness of severalimages taken with varying focuses. Such controls modify the imagecontrast and/or luminosity and/or colour, though do not use a relativesharpness measurement between at least two colours on at least oneregion R of the image.

Furthermore, such known processes as the shade curve and the colourrendering modify the image contrast and/or luminosity and/or colour,though do not use a relative sharpness measurement between at least twocolours on at least one region R of the image.

Such known methods are limited due to the lack of information on thegeometry of the scene. For example, it is difficult to distinguish anaturally dark object from an object poorly lit. As another example, aflashlight is not able to correctly light up several subjects if suchsubjects are at varying distances.

In an embodiment, the function activated comprises the provision of theposition of at least one zone of interest to be considered to aservo-control of exposure and/or of white balance and/or of focussing,such zone of interest being determined by comparing at least tworelative sharpness measurements.

For example, the exposure function may be performed on the part nearestto the capturing apparatus, possibly in combination with anothercriterion, such as elimination of the near object or objects, on theedge of the image (field border).

The servo-control of the white balance may be performed, for example, ona large-scale subject in the centre of the image, possibly to thedetriment of a background lit up differently. As a variant, the methodconsists of determining a close-up part in the image and a far-off part,the white-balance function taking separate measurements on these regionsin order to determine the existence or not of several lightings and toperform distinct compensations for each one of these regions.

If the focussing servo-control is given the position of the interestzone, activation of focussing will be faster and the main subject (zoneof interest) will be able to be followed, even when travelling.

In an embodiment, the function activated comprises provision of asignal, destined for the user, indicating that the image is taken tooclose-up to be sharp.

In an embodiment, the function activated comprises an image resolutionalteration depending on the measured relative sharpness. The image maybe the digital image. The image may also be another image, for examplean image stemming from the same capturing apparatus and captured afterthe digital image.

For example, resolution is reduced when the image is taken at a distancetoo close-up from the capturing apparatus to achieve a sharp image atfull resolution, the final resolution being selected in order to obtaina sharp image.

In an embodiment, the function activated comprises the provision ofdata, or a signal, used for an automatic indexing of the digital image.

For example, if the image comprises subjects or objects at a distancelower than a given limit and of a size exceeding a threshold, indexingmay then consist of providing a signal indicating that it concerns aportrait or a group of persons. The distinction between these twosituations shall be made according to whether the imaged scene comprisesone or several close-up objects or subjects. If the distance of theobjects or subjects exceeds a pre-determined limit, one may thenconsider that the image represents a landscape.

In an embodiment, the function activated comprises the provision ofremote or directional data, in relation to the capturing apparatus, of asubject or an object within the digital image to a sound-capturingdevice.

Thereby, in a camcorder or a cameraphone, it is possible to determinethe main subject or subjects, to determine the distances and/or thedirections of these main subjects and to focus the sound capture on themain subject or subjects, thus eliminating the background noise. Thedirectivity function of the sound capture can be performed using twomicrophones and a de-phasing device between the signals of thesemicrophones.

A particular application of this latest arrangement is, in the case of avideo-conference, the use of a wide-angle image-capturing apparatus andan automatic monitoring of the subject in the process of speaking.

In an embodiment, the function activated includes the parametering ofincreased compression for the background and of a compression for themain subject or subjects, such main subject or subjects being determinedas constituting an image zone complying with the criteria based upon themeasured relative sharpness.

Thereby, in the case of a video-conference, for example, the output canbe minimised, while maintaining a satisfactory visibility of the mainsubject. The latter is determined as constituting the part of the imagethe nearest to the picture-taking apparatus and determined differently,as described in this application.

In an embodiment, the capturing apparatus comprises a sensor havingpixels equipped with coloured filters of at least two types, suchfilters being selected so that their spectral responses entail littleoverlapping.

Under such conditions, the sharpness between two colours can bemaximised, thus optimising the precision of the relative sharpnessmeasurement.

In an embodiment, the capturing apparatus comprises a sensor havingpixels mainly serving to produce the image, and other pixels mainlyserving to measure the relative sharpness.

In an embodiment, the pixels mainly serving to measure the relativesharpness have a spectral response within a spectral band, which entailslittle overlapping with the spectral band of the pixels mainly servingto produce the image.

In an embodiment, the pixels mainly serving to produce the image have aspectral response mainly within the field visible to the human eye, andthe other pixels have a spectral response mainly beyond the fieldvisible to the human eye.

The invention also concerns a sensor thus defined, separate from acapturing apparatus and from the method according to the invention, asdefined above.

The invention also concerns a capturing apparatus comprising such asensor, such capturing apparatus also being able to be used separatelyfrom the method defined above.

The invention also concerns, according to an arrangement which may beused in combination with (or separately from) the arrangements definedabove, a digital image-capturing apparatus comprising a sensorrepresenting, on the one hand, pixels whose spectral response is mainlywithin the field visible to the human eye, and additional pixels havinga spectral response mainly beyond the spectre visible to the human eye,on the other, such sensor being such that the part of the image stemmingfrom these additional pixels represents a sharpness, within at least onerange of distances between the capturing apparatus and the imaged scene,that exceeds the sharpness of the part of the image stemming from thepixels whose spectral response is mainly within the visible field.

The additional pixels may be sensitive to the infrared and/orultraviolet rays. The pixels sensitive to ultraviolet rays may serve toimprove the sharpness for short distances, whereas the pixels sensitiveto infrared rays may serve to improve the sharpness for greaterdistances. By infrared and/or ultraviolet is meant all parts of thespectre beyond or below the visible spectre, notably the near infrared,such as 700 to 800 or 700 to 900 nm, or the near ultraviolet, near by400 nm.

In an embodiment, the capturing apparatus is equipped with a fixed lens,i.e. lacking mechanical elements for focussing.

Under these conditions, focussing can be digitally processed.

In an embodiment, the capturing apparatus is equipped with a lens with avariable focal length without a mobile or flexible focussing element,the relative sharpness between at least two colours on at least oneregion R of the image being variable according to the focal lengthand/or the position of the imaged object in relation to the apparatus.

Thereby, a device equipped with a simpler zoom is obtained, enabling toreduce the size and the cost, and to increase reliability.

The lens with a variable focal length comprises, for example, a singleoptical mobile or flexible unit.

It is known that a zoom is embodied with at least two mobile units, forexample one or two for the focal length and the other for the focussing.Generally-speaking, the focussing and the focal length are separate fromeach other, i.e. when the focal length varies, it is not necessary toalter the focussing. This eliminates the time necessary for focussing.There also exist lenses with variable focal lengths, called varifocals,less costly, in which the focussing must be altered when the focallength varies. Finally, there exist afocal zooms in which two mobileoptical units, linked in a complex manner, are used for the purpose ofvarying the focal length, with focussing being embodied by a third unit.

In an embodiment, the digital image stems from at least two sensors.

For example, each sensor is dedicated to a given colour. It is possible,for example, to use a sensor of the tri-CCD type with a common imaginglens on these sensors.

In an embodiment, the function activated comprises the addition of anobject inside an image and/or the replacement of a part of an imagedepending on the measured relative sharpness on the digital image.

For example, the method enables to add a person next to the mainsubject. As an example, it is also possible to add an object at a givenposition within an image; such object will have the right size withinthe image if one is to take account of the distance from the imagedscene up to such position.

It is also possible to alter the background or even to black it out.

It is also possible to extract a part of the image, such as the mainsubject, and to insert it in another image, whether of the natural orsynthesis type, for example in the context of a game.

It is also possible to add publicity data at a given point and at afixed distance from the scene, for example, behind the main subject.

In an embodiment, the method comprises the capture of a sequence ofimages, the digital image being a part of the sequence and the functionactivated being on at least one other image of the sequence.

Hence, as already described, the estimation of the relative sharpnesscan be taken on images of pre-visualisation prior to picture-taking at alower resolution, while the correction can be made on an image alreadymemorized, for example by using a selection of filters resulting from ameasurement taken from the images of pre-visualisation.

In an embodiment, the function activated comprises the alteration of oneadjustment of the capturing apparatus, namely the focal length, theaperture, the distance for focussing.

Thereby, the picture-taking apparatus may comprise an automaticadjustment programme, such as the aperture being increased if the mainsubject is situated in front of a background, such background thusbecoming blurred. The adjustment programme may also automatically adaptthe aperture to the distance of the subjects of a group so that thedepth of field is adequate enough to make all the subjects in the groupsharp. It is also to be noted that in such a case, a function isautomatically achieved, whereas in the state of the art, it is achievedmanually.

In an embodiment, the function activated comprises the production of analtered raw image.

The digital image is preferably the raw image from the sensor prior to“demosaicing” (i.e. removing the matrix). The digital image may alsohave been processed, for example, undergoing a white balancing. Thedigital image shall preferably not have undergone sub-sampling.

Hence, an optical system, sensor and image-processing means' unit isobtained, thereby producing a raw image representing a better quality orwith specific characteristics, for example, an extension of the depth offield, while maintaining similar characteristics to that of a raw imagedirectly stemming from the sensor, and particularly a compatibility withthe known functional blocks or components performing the function ofconverting a raw image into a visible image (“image pipe” or “imagesignal processor”).

As a variant, the raw image undergoes demosaicing. In an embodiment, thelens of the capturing apparatus represents high longitudinal chromaticaberrations, for example, such as for a given focussing, aperture andfocal length, there exists at least one colour for which the distanceinvolving the best sharpness is lower than

${k\;\frac{f^{2}}{O \cdot P}},$k being a coefficient lower than 0.7, preferably lower than 0.5, f beingthe focal distance, O being the aperture and P having the smallest(among all colours of the image) diameter of the blur spot of an objectpoint situated in infinity.

In an embodiment, the measurement of relative sharpness between twocolours is achieved by comparing the results of a first measurement Mapplied to the first colour and the results of a second measurementapplied to a second colour, each measurement M providing a valuefunction for, on the one hand, sharpness and colour, and for thecontents of the digital image on the other, thus removing the comparisonfrom the digital image contents.

A DEFINITION AND EMBODIMENT EXAMPLE FOR A RELATIVE SHARPNESS MEASUREMENT

The comparison of sharpness is performed using a measurement M on thepixels of the digital image.

The measurement M in a given pixel P, for a channel of a given colour Ccorresponds to the gradient of the variation of C within the P vicinity.It is obtained via the following calculation:

For a given colour C, V(P) is considered as a vicinity of pixel P.

GM is noted as being the average of the amplitude of the gradientswithin the vicinity V(P), and SM as being the average of the amplitudeof the differences between GM and the gradients within the vicinityV(P).

A gradient is calculated through the amplitude of the difference in thevalues of two pixels of a same colour. The gradients within the vicinityV(P) correspond to the gradients implicating a pre-determined number ofpixel couples within the vicinity V(P).

The measurement M to the pixel P having a colour C can be defined by theratio between SM and GM. Thus, a value M (P, C) is obtained.

Such measurement does not itself enable to precisely and completelycharacterise the sharpness of the colour C.

Indeed, it depends on the contents of the image (type of imagedscene=texture, gradation, etc.) within the vicinity V(P) of the pixel P.A frank transition in the imaged scene for a same colour sharpness, willgenerate a higher measurement M than a soft transition within the imagedscene. On natural images, a transition will exist in the same manner ineach colour, thus affecting the measurement M in the same manner amongthe colours. In other words, when a frank transition appears on a colourC, the same type of transition appears on the other colours.

Thereby, the comparison of the measurements M enables to establish therelative sharpness between a colour C1 and a colour C2.

The relative sharpness, between two colours C1 and C2, measured in apixel P can be defined, for example, as a comparison between the twomeasurements M(P,C1) and M(P,C2). Thus M(P,C1)>M(P,C2) implies that C1is sharper than C2.

It is also possible, for example, to use one of the following formulas:M(P,C1)−M(P,C2),M(P,C1)/M(P,C2),

Or any other function F(M(P,C1), M(P, C2) adapted to the comparisonbetween the two measurements.

The relative sharpness in a region R of the image can be defined byusing a measurement M on all the pixels P of the region R.

The relative sharpness in a region R of the image can be the whole rangeor a sub-range of the relative sharpness measured for pixels P of theregion R. It may also be defined as a unique value, such as the sum S ofthe measurements on all the pixels P of the region R for each one of thecolours.

Thereby, for two colours, C1 and C2, it is possible, for example, toconsider that S(C1)>S(C2) implies that C1 is on average sharper than C2in the region R.

It is also possible to use any other function G(S(C1),S(C2)) enablingthe comparison between these two measurements.

In an embodiment, when the function activated consists of determiningthe position of the main subject within the image, the functionactivated further comprises the automatic framing, namely the centeringof the image on the main subject.

The method may be implemented inside an image-capturing or processingapparatus or device. Such apparatuses or devices are included within thegroup comprising: an electronic component, integral or not with asensor, an electronic sub-unit with integrated lens, a sensor andpossibly an image-processing module (“camera module”), or any other formas defined above.

Other specifications and advantages of the invention will be shown inthe description of some of its embodiments, such description beingbacked up by sketches appended hereto, whereupon:

FIGS. 1 a and 1 b, already described, are explanatory diagrams of thelongitudinal chromatic aberration of a converging lens,

FIG. 2, already described, is the colour spectral diagram of an image,

FIGS. 3 a and 3 b are diagrams showing the improvement of a colour'ssharpness using a same sharp colour in accordance with the invention,

FIG. 4 is a diagram showing the improvement of a colour's sharpnessusing different sharp colours linked to distinct regions of an image inaccordance with the invention,

FIGS. 5, 6 and 7 are diagrams showing the improvement of a colour'ssharpness using different sharp colours linked to the whole part of animage in accordance with the invention,

FIG. 8 is a diagram showing the servo-control of an apparatus accordingto a difference in the sharpness between the sharp colour and the colourto be improved in accordance with the invention,

FIG. 9 is a diagram showing the selection of a sharp colour using adistance measured between an object and an apparatus capturing the imageof such object,

FIG. 10 is a diagram showing the reduction in the sharpness of at leastone colour within at least one region of the image,

FIG. 11 is a sketch of an apparatus obtained by the method according tothe invention,

FIG. 12 is a diagram showing the steps of the method according to theinvention,

FIG. 13 shows an adjustment mode in compliance with the invention,

FIGS. 14 a and 14 b are a series of diagrams showing the adjustmentsused in the context of the invention,

FIGS. 15, 15 a and 15 b illustrate a property of an image-capturingapparatus according to the invention and of a conventional apparatus,

FIGS. 16 a to 16 d are diagrams showing the properties of an opticalsystem of an apparatus according to the invention and of a standardapparatus,

FIGS. 17 a and 17 b are sketches showing a selection example of anoptical system for an apparatus in accordance with the invention,

FIGS. 18.1 and 18.2 show the means for implementation of the methodaccording to the invention,

FIGS. 19.1, 19.2 and 19.3 show the steps of the method in accordancewith the invention, according to several embodiment variants, and

FIGS. 20.1 and 20.2 show other embodiments of the invention.

In accordance with the invention, the method described below improvesthe sharpness of at least one colour of a digital image by selectingfrom among the image's colours at least one colour referred to as “sharpcolour” and by reflecting the sharpness of the sharp colour onto atleast one other improved colour, as shown below using FIGS. 3 a and 3 b.

More precisely, FIG. 3 a shows the sharpness (Y-axis 7.2) of two colours13.1 and 13.2 depending on the distance of the objects that theyrepresent in the considered image in relation to the apparatus havingcaptured the image (axis 7.1 of the abscissa).

As previously explained, the sharpness of these two colours varies in adifferent manner depending on such distance, although basically in thisexample, the first colour 13.2 represents a better sharpness than thatof the second colour 13.1 of this same image.

Hence, according to the method complying with the invention, thesharpness of the first colour 13.2 is reflected in order to achieve theimprovement 14 of the sharpness of the second colour 13.1 whichrepresents, after such improvement, an increased sharpness 13.3.

In such example, CA, CO and CN are respectively values representing theimproved colour, the original colour (or colour to be improved) and thesharp colour. In this example, the sharp colour is the first colour. Theoriginal colour and the improved colour correspond to the second colourprior and subsequent to processing.

The sharpness is reflected onto the second colour by using a filter F,according to a formula of the type:CA=CN+F(CO−CN)

Typically, the filter F will demonstrate the particularity of removingthe details from the image on which it is being applied. In order to doso, a linear low-pass filter (or averager) could be used. It is alsopossible to use one of the numerous known non-linear filters having theparticularity of removing the details, like, for example, a medianfilter.

At this stage, it is essential to recall that the human retina isparticularly sensitive, in relation to the details of an image, to thecolour green; hence, the adjustment of optical systems generally aims atachieving extreme sharpness for this colour with regard to a certainfocussing range (cf. for example, pages 30 to 33 of the article, “ColorAppearance Models” by Mark D. Fairchild edited by Addison Wesley).

Hence, according to an observation concerning the invention, an opticaldevice producing images whose sharpness is not satisfactory to the humaneye can represent a satisfactory sharpness for one of its colours, suchas the blue or the red, to which the eye is less sensitive whenconsidering detail.

Typically, for a long-range focussing lens (hyperfocal), whenconsidering an image representing a close-up object and a far-offobject, the sharpness of the far-off object generally appears enhancedwith a green colour, while the sharpness of the close-up object isimproved when considering the blue colour.

It thus appears important to be able to improve the regions of an imageaccording to different sharp colours depending on the relative sharpnessbetween two colours.

Thereby, in an embodiment of the invention, the sharp colour serving toimprove the sharpness of a colour depending on the image's region shallbe selected, such method being described below by way of FIG. 4 whichillustrates an image 10 comprising two regions 11.1 and 11.2.

In these two regions, two colours 8.2 and 8.3 can be found.Nevertheless, the sharpness (Y-axis 7.2) of these colours is such thatin the region 11.1, the colour 8.2 is the sharpest, while in the region11.2, the colour 8.3 is the sharpest.

Henceforth, a colour in the region 11.2 is improved by considering thecolour 8.3 as the sharp colour, while a colour in the region 11.1 isimproved by considering the colour 8.2 as the sharp colour.

At this stage, it should be noted that the regions of an image may ormay not be pre-determined. For example, in the case of a digital imagemade of pixels, a region may be a spatial zone delimited by one orseveral pixels.

Furthermore, it is possible to select a sharp colour in order to improveanother colour by simply comparing the sharpness of one colour inrelation to the others, though at least one other, regardless of anynotion of distance, such as that represented by axis 7.1.

In this case, such an analysis is represented, for example, as a table:

Zone 11.1 11.2 Sharpness 8.2 > 8.3 8.3 > 8.2

In this case, the colour 8.2 is selected as the sharp colour in theregion 11.1, while the colour 8.3 is the sharp colour in the zone 11.2.

Irrespective of the use of regions in an image, it can be advantageousto consider various sharp colours in order to improve a colour in animage, as described below using FIGS. 5, 6 and 7.

More precisely, the diagram in FIG. 5 shows the sharpness (Y-axis 7.2)of two colours 8.2 and 8.3 depending on the distance (7.1) between atleast one object of the captured scene for obtaining the said image andthe capturing apparatus.

It seems that on the range 9.1, the colour 8.3 represents increasedsharpness compared with that of the colour 8.2, whereas for largerdistances (range 9.2), the opposite situation arises.

In such a case, a method complying with the invention may considercolour 8.3 as the sharp colour, serving to correct the sharpness of acolour, within the range of distances 9.1, whereas within the range 9.2,the colour 8.2 is considered as the sharp colour in order to improve acolour stemming from an object of the captured scene for the purpose ofobtaining the image situated at a distance from the capturing apparatus.

Following such corrections, the sharpness of the colours on the imagecan be improved in the direction of a profile, such as shown in diagram6, namely the juxtaposition of the sharpest colours in the image.

It is clear that, in a similar manner to the description in FIG. 4, itis possible to select a sharp colour in order to improve another colourby simply comparing the sharpness of one colour in relation to theothers, though at least one other, regardless of any notion of distance,such as that represented by axis 7.1.

The sharpness curves represented in the figures already described 3 a, 3b, 4, 5 and 6, and subsequently described in 7 to 10, can vary accordingto the geometric position of the region considered of the image and/orof other image-capturing parameters, such as the focal length, theaperture, focussing, etc.

In order to determine the sharpest colour within the meaning of theinvention, there is no need to be aware of the parameters indicatedabove.

In the other cases, and notably for determining the distance accordingto the invention and/or for controlling the depth of field, it isnecessary to be aware of some of the parameters, as well as of thesharpness curves, at least partially or approximately for some values ofsuch parameters.

Furthermore, the choice of the sharp colour can also be determined bythe software activation of at least one image-capturing mode, such as amacro mode, as described hereunder. In such a context, the image may beconsidered as a sole region.

It should be noted that in these FIGS. 5 and 6, a threshold 8.1 isrepresented indicating the level of sharpness required, and above whichthe image is considered as blurred.

In standard processing, shown in FIG. 7, such a threshold 8.1 definesthe depth of field, i.e. the range 9.2 of distances between at least oneobject of the captured scene for obtaining the said image and thecapturing apparatus, such that the image of the object is sharp.

A consequence of the invention is thus to enable an extension of thedepth of field of an optical system, as detailed below by way of FIG. 9.In this figure, the depth of field of a capturing apparatus, initiallylimited by the sharpness of the colour 8.2 and the sharpness threshold8.1, is increased by using a second colour 8.3 representing asatisfactory sharpness (below threshold 8.1) on a new range of distancesbetween at least one object of the captured scene for obtaining the saidimage and the capturing apparatus.

Concretely, such an application is implemented in fixed-focusphotographic apparatus, such as cameraphones. Indeed, the opticalconception of these apparatus allows for a sharpness range for longdistances, up to several tens of centimetres at the best, on the basisof a green colour, similar to the colour 8.2 of FIG. 5.

Furthermore, the blue colour not focussing in the same manner, it canrepresent sharpness at shorter distances where the green colour may not,similarly to the colour 8.3.

Henceforth, the invention enables to increase the sharpness of aclose-up image of a cameraphone by attributing the sharpness of the bluecolour to the green colour, and to the other colours, consequentlyincreasing the depth of field of the apparatus.

In an embodiment of the invention, shown by way of FIG. 8, moreespecially adapted to a capturing apparatus equipped with an autofocusfunction, the method determines a servo-control instruction for theconsidered capturing apparatus using the sharpness of at least twocolours of the captured image, in such a manner that focussing isachieved in fewer steps and thus more rapidly.

For example, a distance 17.1, between at least an object of the imagedscene and the optical system 1 capturing the image, may be determinedusing the various levels of sharpness (Y-axis 7.2) of the colours 8.2and 8.3 used in the region 11.3 relating to the image of the object.

Knowing such a distance between the object 4 and the system 1, it isthus possible to determine a servo-control instruction 5 for thecapturing apparatus 6. Such FIG. 8 shall be described below in moredetail.

According to another embodiment of the invention, shown by way of FIG.10, the sharpness is reduced by at least one colour in at least oneregion of the image.

Macro Application

We are now going to describe, by referring to FIGS. 5, 6 and 7, anembodiment and system according to the invention, more especiallyadapted to the embodiment of a Macro function without requiring aspecific mechanical device for a known image-capturing apparatus. Amacro function is destined to enable the image embodiment of objectsclose to the capturing apparatus within a pre-determined range ofdistances, called range of macro distances 9.1, on the apparatus.Usually, a capturing apparatus enables to move all or part of the lensin order to embody the macro function. The method or object system ofthe invention enables to do away with such movement.

According to the invention, the sharpest colour for the range of macrodistances 9.1 shall be pre-determined, for example, through themeasurement of the sharpness 8.2 and 8.3 of the colours of the digitalimages obtained by the capturing apparatus for each colour, by embodyingthe digital images using objects located at different distances from thecapturing apparatus. The sharpest colour (FIG. 5) is the onecorresponding to the measurement 8.3. Such pre-determination can beembodied definitively, for example, when designing the apparatus (or aseries of apparatus).

Thereafter, when using the apparatus, upon activation of the Macrofunction, the sharpness of the sharp colour thus determined shall bereflected onto the other colours, as described above. When the Macrofunction is not activated, the sharpness of the digital image can becalculated by a standard method or by using the method according to theinvention, as applied to the range of distances 9.2.

Thereby, a macro function is achieved, compatible with a fixed focussinglens without any mobile mechanism, thus not altering the overalldimensions of the image-capturing apparatus, nor adding hardware costs.The macro mode may thus be activated via software within the apparatusor within any other image-processing device. Such software activationmay be performed in a standard manner prior to image capture, but alsoafter such capture and on a local or remote device of the capturingapparatus. According to a variant, activation of the macro mode may bedone automatically, for example by determining the sharpest imagebetween the image generated in normal mode and the image generated inmacro mode.

The macro function embodied according to the invention is alsobeneficial to an apparatus comprising variable parameters when capturingthe digital image and having an influence on the sharpness of thecolours, notably a capturing apparatus with a zoom, and/or a lens withvariable focussing and/or a variable aperture. Thus, the sharpnesscurves 8.2 and 8.3 shall be used, corresponding to the value of thevariable parameters according to the digital image.

The addition of the macro function enables to take shots of bar codes,business cards or handwriting containing text and/or sketches, by usingan image-capturing apparatus, notably a telephone or a photo apparatus.

Depth of Field Extension Application

We are now going to describe, by referring to FIGS. 4, 5, 6 and 7, anembodiment and system according to the invention, more especiallyadapted to the extension of the depth of field without requiring aspecific mechanical device for a known image-capturing apparatus. Thedepth of field corresponds to the range of distances, between theobjects of the scene and the image-capturing apparatus, enabling toobtain a sharp digital image. Usually, a capturing apparatus has alimited depth of field and the lower it is, the greater the lensaperture.

According to the invention and as represented in FIG. 4, the digitalimage is decomposed into regions 11.1 and 11.2, for example, into squareregions corresponding to 9 sensitive elements next to the sensor or,more generally, into regions corresponding to X by Y sensitive elementsor into regions of a pre-determined shape or calculated according to thedigital image. For each region shall be selected the sharpest colour,for example, like the colour corresponding to the lowest value among thevalues obtained by calculating a gradient for each colour using greylevels corresponding to the colour and the region considered. In FIG. 4,the colour corresponding to the curve 8.3 is sharper for the region11.2, while the colour corresponding to the curve 8.2 is sharper for theregion 11.1.

Thereby, for each region, the sharpness of the sharp colour thusselected is reflected onto the other colours.

By referring to FIG. 5, the digital image of close objects—having adistance 5 on the capturing apparatus within the range of distances9.1—can be seen as being sharp for the colour corresponding to the curve8.3 (for example the blue colour), while it is less so for the colourcorresponding to the curve 8.2 (for example the green). It is alsopossible to see that the digital image of far-off objects—havingdistances on the capturing apparatus comprised within the range ofdistances 9.2—is sharp for the colour corresponding to the curve 8.2,while it is less so for the colour corresponding to the curve 8.3. Theeye being much more sensitive to sharpness within the green than withinthe blue, it will perceive a sharpness corresponding to the curve 8.5 ofFIG. 7. If 8.1 corresponds to the threshold of sharpness for the eye,the image will only be sharp for the objects located at a distance fromthe capturing apparatus comprised within the range 9.2. FIG. 6represents, via the curve 8.4, the sharpness obtained in each colourafter employing the method according to the invention: the blue hasenabled to obtain a better sharpness than the threshold 8.1 for the nearobjects, located in the range of distances 9.1, whereas the green hasenabled to obtain a better sharpness than the threshold 8.1 for thedistant objects, located in the range of distances 9.2. Hence, a sharpdigital image is achieved for all the colours within a large range ofdepth of field.

This is an example consisting of selecting the sharpest colour accordingto a pre-determined rule, namely choosing the sharpest colour in eachregion.

Thereby, the depth of field is increased without increasing the cost,the complexity or the overall dimensions of the optics and/or with noneed to change the exposure, thus to reduce the aperture, to increasethe noise level or to increase the movement blur.

The increase in the depth of field embodied according to the inventionis notably beneficial to the fixed lenses, namely to telephones. Theincrease in the depth of field enables to take not only shots of barcodes, business cards or handwriting containing text and/or sketches,but also of portraits or landscapes, by using an image-capturingapparatus, notably a telephone or a photo apparatus. This is possiblewithout using the costly autofocus or macro functions. Furthermore, thisfunction, compared with a mechanical macro function, is achievedentirely automatically without intervention by the user.

The increase in the depth of field embodied according to the inventionis also beneficial to an apparatus comprising variable parameters whencapturing the digital image and having an influence on the sharpness ofthe colours, notably a capturing apparatus with a zoom, and/or a lenswith variable focussing and/or a variable aperture. Thus, the sharpnesscurves 8.2 and 8.3 shall be used, corresponding to the value of thevariable parameters according to the digital image.

The method and function according to the invention thus enables toselect or design, as subsequently described by way of FIGS. 11 to 17 b,at the time of the capturing apparatus being designed, a lens with amore limited number of focussing positions, which has the advantage ofreducing the lens-design constraints and thus of reducing the coststhereon. This also has the advantage of allowing faster and less-costlyfocussing by reducing the precision required for the servo-controlmechanism.

For example, in order to obtain a large depth-of-field lens, it ispossible to choose or design a lens having the specification of beingequipped with the broadest union of sharp-distance ranges for each oneof the colours.

For example, in order to obtain a large aperture lens, it is possible tochoose or design a lens having the specification of being equipped witha single sharp colour within each one of the ranges of distances, andsuch that the union of sharp-distance ranges for each one of the colourscorresponds to the depth of field desired.

In another example, it is also possible to optimise both the aperture ofthe apparatus and the image's depth of field.

A method and a function are also achieved enabling to reduce thelongitudinal chromatic aberrations of a digital image.

A method and function is also obtained enabling to increase thesharpness of an image without knowing which capturing apparatus was usedto produce it.

Application on the Distance Measurement for Objects of a Scene Using aSingle Image

We are now going to describe, by referring to FIG. 8, an embodiment andsystem according to the invention, more especially adapted to measuringthe distance of the objects of a scene using a single image withoutrequiring a range-finding hardware measuring device. The method thusenables to obtain an estimation of the distance of the objects presentin each region of the digital image.

Usually, a capturing apparatus uses a hardware device for measuring thedistance of the objects of a scene based on a laser, an infrared or apre-flash mechanism, amongst others.

According to the invention and as represented in FIG. 8, the digitalimage is decomposed into regions 11.3, for example, into square regionscorresponding to 9 sensitive elements next to the sensor or, moregenerally, into regions corresponding to X by Y sensitive elements orinto regions of a pre-determined shape or calculated according to thedigital image. Thereby, for each region 11.3, the sharpness of at leasttwo colours is measured; such measured values or measured relativevalues 16.1 and 16.2 are reported onto the corresponding sharpnesscurves 8.2 and 8.3 of the capturing apparatus. Hence, a distance 17.2 isobtained, corresponding to an estimation of the distance 17.1 betweenthe part of the object 4, represented in the region 11.3, and thecapturing apparatus.

The distance measurement embodied according to the invention is notablybeneficial to the fixed lenses, namely to telephones.

The distance measurement embodied according to the invention is alsobeneficial to an apparatus comprising variable parameters when capturingthe digital image and having an influence on the sharpness of thecolours, notably a capturing apparatus with a zoom, and/or a lens withvariable focussing and/or a variable aperture. Thus, the sharpnesscurves 8.2 and 8.3 shall be used, corresponding to the value of thevariable parameters according to the digital image.

The method thus enables to obtain an estimation of the distance of theobjects present in each region of the digital image. This enables:

-   -   to build a real-time and low-cost range-finder device by way of        a sensor and of a standard lens which produces an image and the        remote data correlated to the image; usually, several shots are        necessary or a specific hardware device is required and the        association image/remote data is complex;    -   for example, the distance is displayed in real-time on the        image,    -   for example, the remote data enables to guide a robot,    -   to accelerate the focussing of capturing apparatuses with        variable focussing or focal lengths: it is indeed possible to        determine, using a single image, the servo-control instructions        to be applied in order to obtain the desired focussing, for        example on the central subject or in a focussing zone selected        by the user,    -   to take account of the distance of the various objects of a        scene in order to adjust the strength of a flashlight and        notably of the main subject or of the subject in the focussing        zone,    -   to take account of the distance of the various objects of a        scene for the auto-exposure function of the capturing apparatus,        in order, for example, to enhance the main subject or the        subject in the focussing zone selected by the user for a        portrait,    -   to automatically define the main subject without having to ask        the user to define it.

Application on the Depth of Field Control, Irrespective of the Exposure,

We are now going to describe, by referring to FIGS. 4, 5, 6 and 7, anembodiment and system according to the invention, more especiallyadapted to the control of the depth of field without requiring aspecific mechanical device for a known image-capturing apparatus. Themethod thus enables to obtain a sharp image for objects situated awayfrom the capturing apparatus, corresponding to a range of sharpness anda blurred image for the other objects. Usually, a capturing apparatushas a limited depth of field and the lower it is, the greater the lensaperture; hence, the depth of field and the exposure are linked in sucha manner that a choice has to be made when in low lighting between depthof field, noise and movement blur. According to the embodiment, it ispossible to separately control exposure and depth of field.

According to the invention and as represented in FIG. 4, the digitalimage is decomposed into regions 11.1 and 11.2, for example, into squareregions corresponding to 9 sensitive elements next to the sensor or,more generally, into regions corresponding to X by Y sensitive elementsor into regions of a pre-determined shape or calculated according to thedigital image. For each region shall be selected the sharpest colour,for example, like the colour corresponding to the lowest value among thevalues obtained by calculating a gradient for each colour using greylevels corresponding to the colour and the region considered. In FIG. 4,the colour corresponding to the curve 8.2 is sharper for the region11.2, while the colour corresponding to the curve 8.3 is sharper for theregion 11.1.

Thereby, for each region, the sharpness of the sharp colour thusselected is reflected onto the other colours, as previously described.As seen above, a sharp digital image is thus obtained for all thecolours in a large range of depth of field.

In order to determine the distance between the capturing apparatus andthe objects of the captured scene in the region of the digital image,shall be used:

-   -   either, as previously described, the sharpness of at least two        colours for each region, or    -   another more precise distance-measuring method or device.

Thereby, it is possible to reduce the sharpness, for example, using aGaussian filter, or using a filter simulating a bokeh, in the regionsand/or in the parts of the field containing the objects located atdistances beyond the range of desired sharpness. For example, for aportrait, a blurred background can be obtained, thus enhancing the facewithout requiring a wide-aperture lens. For example, for a landscape, avast depth of field can be obtained, except possibly for isolatedobjects in the corners, which may hinder comprehension of the image. Forexample, for a scene comprising close-up objects in the corner as aresult of poor framing, such close-up objects can be blurred. Forexample, the choice of the depth of field can be left to the discretionof the user, either within the apparatus or on a computer duringpost-processing.

Thereby, the depth of field is controlled without needing to changeexposure, thus without altering the aperture, nor increasing the noiselevel or increasing the movement blur.

The control of the depth of field embodied according to the invention isnotably beneficial to the fixed lenses, namely to telephones. Thecontrol of the depth of field enables to take not only shots of barcodes, business cards or handwriting containing text and/or sketches,but also of portraits or landscapes, by using an image-capturingapparatus, notably a telephone or a photo apparatus. This is possiblewithout using the costly wide-aperture lens device. Furthermore, thisfunction can be achieved entirely automatically without intervention bythe user.

The control of the depth of field embodied according to the invention isnotably beneficial to an apparatus comprising a mobile lens, notably azoom. A well-informed connoisseur can thus directly or indirectly takecontrol, irrespective of the depth of field and the exposure.

FIG. 11 is a sketch illustrating the architecture of an image-capturingor reproducing apparatus.

Such an apparatus, for example, for capturing images, comprises, on theone hand, an optical system 122, notably with one or several opticalelements, such as lenses, destined to form an image on a sensor 124.

Although the examples mainly concern a sensor 124 of the electronictype, such sensor may be of another type, for example, a photographicfilm in the case of an apparatus known as “argentic”.

Such an apparatus also comprises a servo-control system 126 acting onthe optical system 122 and/or on the sensor 124 in order to performfocussing so that the image plane is captured on the sensor 124, and/orso that the quantity of light received on the sensor is optimal due toadjustment of the exposure and/or aperture time, and/or so that thecolours obtained are correct, by performing a white-balanceservo-control.

Finally, the apparatus comprises digital image-processing means 128.

As a variant, such digital image-processing means are separate from theapparatus 120. It is also possible to plan a part of theimage-processing means within the apparatus 120 and a part outside theapparatus 120.

The digital processing of the image is performed after image-recordingby the sensor 124.

An image-reproducing apparatus represents a similar structure to animage-capturing apparatus. Instead of sensor 124, an image-generator124′ is provided, thus receiving the images from digitalimage-processing means 128′ and providing the images to an opticalsystem 122′, such as an optical projection system.

In the next part, when mentioning clarity of exposure, onlyimage-capturing apparatus are being referred to.

According to one of its aspects, which may be used separately from theaspects previously described, the invention, using the capacity of themeans 128, 128′, consists of digital image-processing for determining orselecting the parameters of the optical system 122, 122′ and/or of theimage sensor or generator 124, 124′ and/or of the servo-control system126.

In diagram in FIG. 12 are represented the level of performances that canbe attained with each one of the components of the apparatus when theyare associated with digital image-processing means. Such levels areillustrated by the discontinued line 130 for the optical system, thediscontinued line 132 for the sensor, the discontinued line 134 for theservo-control, and the discontinued line 136 for the apparatus.

Using such levels of performance that can be obtained with digitalimage-processing means, it is possible to select the performance levelsfor each one of the components of the apparatus which are, prior toprocessing, considerably lower than the levels of performance obtainedafter application of the processing means. Thereby, the levels ofperformance of the optical system can be set at level 130′, and thelevels of performance of the sensor and of the servo-control system canbe set respectively at levels 132′ and 134′.

Under these conditions, failing digital processing, the level of theperformances of the apparatus would be at the lowest level, for examplelevel 136′ corresponding to the lowest level 130′ for the opticalsystem.

The digital image-processing means are preferably those described in thefollowing documents:

-   -   Patent application EP 02751241.7, entitled: “Method and system        for producing formatted data related to defects of appliances in        a series of appliances and formatted data destined for        image-processing means”.    -   Patent application EP 02743349.9 for: “Method and system for        modifying the qualities of at least one image originating from        or destined to a series of appliances”.    -   Patent application EP 02747504.5 for: “Method and system for        reducing the frequency of updates for image-processing means”.    -   Patent application EP 02748934.3 for: “Method and system for        correcting chromatic aberrations of a colour image produced by        an optical system”.    -   Patent application EP 02743348.1 for: “Method and system for        producing formatted data related to geometric distortions”.    -   Patent application EP 02748933.5 for: “Method and system for        providing, according to a standard format, formatted data to        image-processing means”.    -   Patent application EP 02747503.7 for: “Method and system for        calculating an image transformed using a digital image and        formatted data relating to a geometric transformation”    -   Patent application EP 02747506.0 for: “Method and system for        producing formatted data related to defects of at least one        apparatus in a series, notably to blur”.    -   Patent application EP 02745485.9 for: “Method and system for        modifying a digital image taking into account its noise”.    -   Patent application PCT/FR 2004/050455 for: “Method and system        for differentially and regularly modifying a digital image by        pixel”.

Such digital image-processing means enable to improve the image qualityby activating at least one of the following parameters:

-   -   The geometric distortions of the optical system. It is recalled        that an optical system can distort the images such that a        rectangle can be deformed into a cushion, with a convex shape        for each one of its sides, or into a cylinder with a concave        shape for each one of its sides.    -   The chromatic aberrations of the optical system: if a targeted        point is represented by three coloured spots having precise        positions one in relation to the other, the chromatic aberration        is translated by a variation in the position of such spots one        in relation to the other, the aberrations generally being that        much more significant the more one moves away from the centre of        the image.    -   The parallax: when implementing an adjustment by deformation or        movement of an optical element of the optical system, the image        obtained on the image plane can be moved. The adjustment is, for        example, an adjustment of the focal length or a focussing        adjustment.

Such defect is illustrated by FIG. 13 in which an optical system 140 isrepresented with three lenses in which the centre of the image occupiesthe position 142 when the lens 144 occupies the position represented bya continual line. When the lens 144 moves into the position 144′,represented by discontinued lines, the centre of the image adopts theposition 142′.

-   -   Depth of the field: when the optical system is focussed on a        determined object plane, not only the images of this plane        remain sharp, but also those of the objects close to such plane.        “Depth of field” refers to the distance between the nearest        object plane and the farthest object plane for which the images        remain sharp.    -   Vignetting: the luminosity of the image is generally maximal at        the centre, progressively lowering as one moves away from the        centre. Vignetting is measured, in percentage, by the difference        between the luminosity in a particular point and the maximal        luminosity.    -   The lack of sharpness of the optical system and/or the image        sensor and/or generator is measured, for example, by the BXU        parameter, such as is defined above.    -   The noise of the image is generally defined by its difference        type, its shape and the size of the noise spot and its        colouring.    -   The moiré phenomenon is a deformation of the image which occurs        in the event of spatial high frequencies. The moiré is corrected        by the parametering of anti-aliasing filters.    -   The contrast is the ratio between the highest and the lowest        luminosity values of the image for which the details of the        image still remain visible.

As represented in FIGS. 14 a and 14 b, it is possible to improve thecontrast (FIG. 14 a) of an image, i.e. to extend (FIG. 14 b) the rangeof luminosities on which detail can be distinguished. Such extension isperformed by notably using a contrast and noise correction algorithm.

Referring to FIG. 15, we are now going to describe an embodimentenabling to harmonize the sharpness within the image field.

First, it is recalled that the image surface of an object plane does notconstitute a perfect plane, but represents a curve, known as the fieldcurve. Such curve varies depending on diverse parameters, including thefocal length and focussing. Thereby, the position of the image plane 150depends upon the zone on which focussing is performed. In the exampleshown in FIG. 15, the plane 150 corresponds to focussing at the centre152 of the image. In order to focus on a zone 154 near the edge of theimage, the image plane 156 is located nearer to the optical system 122than the image plane 150.

In order to simplify the focussing servo-control system, the image planeis placed in a position 158, mid-way between the positions 154(corresponding to focussing on a zone near the edge of the image) and150 (corresponding to focussing on a zone at the centre of the image).The uniting of the digital image-processing means 128 with the focussingservo-control 126 enables to limit the movement of the plane 158 forfocussing, thus reducing the energy consumption of the servo-controlsystem and enabling to reduce the volume of its components.

The diagram in FIG. 15 a represents the blur properties with a standardservo-control focussing system wherein the maximum sharpness is obtainedat the centre of the image. Thereby, on such diagram in FIG. 15 a, theabscissa represents the field of the image and the ordinates representthe blur value expressed in BXU. Using such standard servo-controlsystem, the blur measurement is, at the centre, by 1.3 and, at the edgeof the image, by 6.6.

FIG. 15 b is a similar diagram to that of FIG. 15 a, showing theproperties of a servo-control for an apparatus embodied according to theinvention, on the assumption that the digital image-processing meansenable to correct the blur up to a BXU value equal to 14. The curverepresented in this diagram in FIG. 15 b thus represents, at the centreof the image, a BXU value=2.6, with the BXU value lowering as one movesaway from the centre, before increasing once again up to a value of 4near the edge of the image. It is recalled that such value is the limitfor enabling correction of the blur by digital processing means.Thereby, a sharp image can be obtained across the entire image field,whereas this is not so using an apparatus equipped with a standardsystem.

In an embodiment, the digital image-processing means comprise means forimproving the sharpness, such that they enable to refrain from using afocussing servo-control.

As a comparable example, the diagrams in FIGS. 16 a, 16 b, 16 c and 16 dshow the specifications of an apparatus obtained according to the stateof the art and those of an apparatus obtained using the method accordingto the invention.

The standard device is a digital photographic apparatus integral with amobile telephone having a VGA sensor, i.e. a resolution of 640×480,without a focussing system.

The standard apparatus has an aperture of 2.8, whereas the apparatusobtained using the method according to the invention has an aperture of1.4.

FIG. 16 a, which corresponds to the standard apparatus, is a diagram onwhich the abscissa represents the percentage of the image field, itsorigin corresponding to the centre of the image. The ordinate representsthe vignetting V.

FIG. 16 b is a similar diagram for an apparatus obtained according tothe invention.

In the schema of FIG. 16 a (standard apparatus), the vignetting attainsthe value of 0.7 at the edge of the image, whereas in the diagram inFIG. 16 b can be seen the optical system of the apparatus according tothe invention, representing a vignetting considerably more significant,i.e. approximately 0.3. The correction limit for the algorithm used is0.25. In other words, due to the correcting algorithm, it is possible toemploy considerably more significant vignetting optics.

FIG. 16 c is a diagram representing the blur ordinates, expressed inBXU, in accordance with the image field (represented in abscissa) for astandard apparatus. Using such standard apparatus, the blurspecification is 1.5 at the centre and 4 at the edge of the image.

The diagram in FIG. 16 d also represents the blur for the optics of theapparatus, obtained using the method according to the invention. In thediagram in FIG. 16 d, the field of image is also represented in abscissaand the blur is represented in ordinates expressed in BXU. Such diagramin FIG. 16 d shows that the blur at the centre of the image isapproximately 2.2. It is, therefore, higher than the blur of the diagramin FIG. 16 c. However, on the edges, a blur has been chosen in theregion of 3, taking account of the correction algorithm limit.

In other words, surprisingly, a gradation lens was chosen with regard tothe sharpness at the centre, even though it is possible to obtain thesame results as when using the standard apparatus with, in addition, agreater aperture. It is also to be noted that on the edges, the opticsof the apparatus according to the invention represent a similar qualityto that of the standard optics, such result being possible due to thevignetting gradation in relation to a standard lens.

The diagrams in FIGS. 17 a and 17 b represent the specifications of thevarious optical systems from among which the selection has to be made inorder to embody a capturing apparatus by using the method according tothe invention.

In the example represented in FIG. 17 a, the optical system provides animage spot 1100 with small dimensions. Such system shows a modulationtransfer function (MTF) represented by a diagram where the spatialfrequencies are in abscissa. The value of the shut-off frequency is fc.The MTF function comprises a step 1110 within the vicinity of the nilfrequencies and a part rapidly decreasing towards the fc value.

The optics represented by the schema in FIG. 17 b show an image spot1114 having considerably larger dimensions than the image spot 1100,with its MTF showing the same fc shut-off frequency as in the case ofFIG. 17 a. However, the variation of this MTF depending on the spatialfrequency is different: such frequency reduces in a relatively evenmanner from its origin down towards the shut-of frequency.

The choice of an optical system is based on the fact that the correctionalgorithm of the modulation transfer function is effective as from avalue of 0.3. Under such conditions, we note that with the optics inFIG. 17 b, a correction is obtained enabling to increase the MTF up to avalue of f₂, for example, approximately 0.8 fc, whereas with the opticsin FIG. 17 a, the correction is only possible up to a frequency f₁ inthe range of 0.5 fc.

In other words, with a correction algorithm, the optics represented inFIG. 17 b provide more detail than the optics represented in FIG. 17 a,and this despite the fact that the image spot is of greater dimensionsthan in the case of FIG. 17 a. Hence, we will choose the opticcorresponding to FIG. 17 b.

Application on the Increase of the Depth of Field

We are now going to describe an embodiment variant of the method forwhich the sensor and/or the optics system are more specifically adaptedto the increase in the depth of field.

The CMOS or CCD standard sensors are often sensors formed using a mosaicof pixels, referred to as “Bayer”. The Bayer mosaic consists of asuccession of 2×2 pixels, formed by 2 green pixels (i.e. a photositesensitive to the light within a spectral range around 550 nm), by a redpixel (spectral range around 600 nm) and by a blue pixel (spectral rangearound 450 nm). The spectral ranges are shown in FIG. 2.

Depending on the sensors, the spectral bands of the green colour, of thered colour and of the blue colour differ, showing an overlapping more orless significant. Significant overlapping among these three bands hasthe effect of reducing the sensitivity of the sensor to colours (itbecomes “colour blind”), but increases its overall sensitivity to thelight and conversely.

Significant overlapping among these three bands also reduces thesharpness differences between the colours, thus notably reducing therange of distances for which at least one of the three colours is sharp.

Hence, advantageously, according to the invention, it is possible toadapt the spectral bands, for example, by reducing their overlapping soas to increase the range of distances for which at least one of thethree colours is sharp.

Such adaptation could also be conducted in conjunction with the designof the optics and, depending on the constraints weighing on the digitalprocessing of the image.

Description of a Sensor Optimising the Method According to the Invention

In an embodiment variant of the method, the sensor and/or the opticssystem are more specifically adapted to applications enabling to provideprecise distance indications for the imaged objects.

In this embodiment variant, we shall use a Bayer pixel mosaic.

It is common that the sensors represent a significant number of pixelsproviding aberrant digital values. These pixels are commonly called“burned pixels,” (or “pixels morts” i.e. “dead pixels” in French).Thereby, image-generating digital processing contains a filtering stepfor these aberrant values in order to erase, from the generated image,the aberrant values of these pixels to make them invisible.

The precision of the distance measurements according to the methodnotably depend upon the variation of the relative sharpness depending onthe distance. Such variation depends upon the amount of chromaticaberration that may be obtained with the capturing system (sensor andoptics). Having said that, the spectral frequency range for the visiblelight, hence the light available for photography, is relativelyrestricted: approximately 400 nm to 700 nm. Thereby, the relativesharpness variation depending on the distance thus becomes limited whenusing a standard Bayer sensor.

Several possible ways exist in which to alter a sensor beyond suchlimitation. A simple manner consists of using a different spectral bandin addition to the standard colours of red, green and blue, e.g. a 800nm-900 nm band or any other band above and/or below the visible spectre.The pixels sensitive to such fourth spectral band will not necessarilybe useful to the rebuilding of the visible image, but will mainly servefor estimating the distance of the objects in comparison to the relativesharpness on this fourth spectral band with one or several of the threestandard colours.

Hence, it will be possible to advantageously arrange the pixels in thefollowing manner: by departing from a standard red, green, blue Bayerlayout, all the N×M pixels and several other pixels will be substitutedby pixels that are sensitive within such fourth spectral band. Byselecting N and M with a rather large factor (for example, 64 each) andby substituting 9 of the pixels, we can thus ensure that onlyapproximately 1 pixel out of 1000 in the standard Bayer mode isaffected. Hence, during the building of the image, such pixels will beconsidered as “burned pixels” with their values being filtered.

Thereby, a photographic apparatus is obtained, enabling to provide moreprecise distance indications of the imaged objects every N×M pixels ofthe image.

Description of a Second Sensor Optimising the Method According to theInvention

In another embodiment, shown in FIG. 20.2, we depart from a standardBayer layout in which are provided three pixels R, G, B and one pixel Ucorresponding to a part of a UV or infrared spectral band. By infraredand/or ultraviolet is meant all parts of the spectre beyond or below thevisible spectre, notably the near infrared, such as 700 to 800 or 700 to900 nm, or the near ultraviolet, near by 400 nm. Such pixel U is used toimprove the sharpness of the visible colours as shown in the diagram ofFIG. 20.1.

On this diagram are noted: the distances “d” of the objects imaged witha capturing apparatus, in abscissa, and the diameter “D” of the blurspot, in ordinates. The curves, 20.3, 20.4, 20.5 and 20.6 represent thevariation of the diameter “D” depending on the distance “d” for,respectively, the red “R”, the green “G”, the blue “B” and theultraviolet “U”. The right 20.7 represents the threshold of sharpnessdefining the depth of field.

Hence, the distance “d1” represents the limit of the depth of field fora capturing apparatus comprising “RGB” pixels and not U pixels, whileusing the method for improving sharpness according to the invention. Thedistance “d2” represents the limit of the depth of field obtained with acapturing apparatus comprising the sensor represented in FIG. 20.2 andusing the method for improving sharpness according to the invention. The“U” pixels serve only to reflect the sharpness of the “U” colour ontothe “RGB” colours for those objects located between the distances “d1”and “d2”. Hence, the final image will only comprise the three “RGB”colours (or any other known visible colour span).

As a variant, pixels sensitive to the near infrared will be added inorder to improve the sharpness on greater distances.

Increase of the Longitudinal Chromatic Aberrations

According to the invention, advantage is taken of the existence ofvariations in the relative sharpness between two colours depending onthe distance of the objects. Hence, we will be able to design opticsrepresenting relative sharpness between the three very different colourplanes depending on the distance. Such optics are said to represent highlongitudinal chromatic aberrations.

In a practical sense, optics could, for example, be designed so that ona wide range of distances: the smallest of the spot diagram diameters(diameter of the blur spot) from among the three colours is below afirst pre-determined threshold, and the biggest of the spot diagramdiameters from among the three colours is below a second pre-determinedthreshold. Alternatively, the BXU value could be used instead of thediameter of the spot diagram.

The two thresholds are determined according to, for example, thecapacities and constraints of the digital processing for generating theimage, on the one hand (like, for example, the size of the filter “F”described below), and the specifications of the sensor, on the other.

The figures represent an example of the BXU measurements (Y-axis) pourthe three RVB colour planes, depending on the distance (axis inabscissa) for a lens designed in this manner. The values shown are thoseat the centre of the image field. For each point of the image field,various, although similar, curves can be measured. S1 and S2 designatethe two thresholds described above. The range of distances complyingwith the above two criteria is thus, for this lens, approximately 12cm-infinity (d1->infinity in FIG. 18), which implies that it is possibleto rebuild a sharp image for scenes imaged within such range ofdistances.

Using a standard lens would have resulted in three curves, near to thecurve of the red R colour, and thus in an optics system only enablingthe rebuilding of sharp images for objects located at far-off distances,25 cm-infinity (d2→infinity).

Hence, it is also possible to use a lens having longitudinal chromaticaberrations, such as for a given focus, aperture and focal length, wherethere exists at least one colour for which the distance involving thebest sharpness is lower than

$k\;\frac{f^{2}}{O \cdot P}$k being a coefficient less than 0.7, preferably lower than 0.5, f beingthe focal distance, O being the aperture and P having the smallest(among all colours of the image) diameter of the blur spot of an objectpoint situated in infinity.

Application on the Automatic Resolution Adaptation

We are now going to describe an embodiment variant of the inventionenabling to automatically adapt the resolution of the image to thepossible blur linked to a shot beyond the depth of field of thecapturing apparatus.

When the imaged scene is too close-up (below the depth of field), theimage is blurred, i.e. the spot diagram (the blur spot) occupies anadditional spot having X pixels in diameter, X being a pre-determineddiameter defining the limit of the depth of field. A digitalsub-sampling of the image (zoom out), shall reduce the size of the blurspot by a factor dependant upon the type of sub-sampling used, thoughtypically within the size-range of the considered sub-sampling factor.Thereby, a sharp image will be able to be generated using the digitalimage, though of lower resolution by selecting the sub-sampling factorso that the blur spot is lower, once the image has undergonesub-sampling, at given threshold.

As a variant, in order to minimise calculations, we shall begin bysub-sampling, as described above, before increasing the sharpnessaccording to the invention.

Application on the Alteration of Sharpness—Filtering

As for the distance measurement (FIG. 8), it is possible to extract theexpected sharpness from each colour of the digital image using therelative sharpness measurement between two colour planes.

According to an embodiment of the invention, the process includes asharpness alteration for each pixel of the zone Z′, by way of a filtermixing the pixel values within a predetermined vicinity of each pixel,the filter parameters depending upon the measured relative sharpness.

Indeed, an image-capturing device equipped with a lens will showdifferent sharpness depending on the colour planes and according to thedistance of the imaged objects. The fact of the sharpness (or the blur)being dependent upon the distances of the imaged objects makes itimpossible to increase sharpness by using a pre-determined process, suchas a filtering of pre-determined sharpness.

An embodiment variant of the invention consists in selecting or inadapting the sharpness filters to the measured relative sharpness.

A particular case, already described, of the adaptation of the filteringor alteration of sharpness consists of reflecting the sharpness of thesharp colour onto at least one other improved colour, such reflectionbeing achieved by using a calculation of the type CA=CN+F(CO−CN), whereCA represents this improved colour, CO represents the improved colourprior to processing, CN represents the sharp colour and F represents afilter, namely a low-pass filter.

However, in a more general manner, a sharpness filter involving all (ora sub-group of all) colours could be used. Hence, in the case of adigital image RGB (or RVB) for processing the value of a pixel P, thefilter M may alter the value of pixel P, depending on the values of thepixels within a vicinity of the pixel P on all three colours.

For example, by noting RN, GN, BN, the digital data relating to the red,green and blue colours of the digital image, and RA, GA, BA, the digitaldata relating to the colours of the improved image, it is possible toselect the filter M, such as an operator undertaking the followingoperations:GA=GN+c _(—) GG*M _(—) GG(GN)+c _(—) GR*M _(—) GR(RN)+c _(—) GB*M _(—)GB(BN)RA=RN+c _(—) RG*M _(—) RG(GN)+c _(—) RR*M−RR(RN)+c _(—) RB*M _(—) RB(BN)BA=BN+c _(—) BG*M _(—) BG(GN)+c _(—) BR*M−BR(RN)+c _(—) BB*M _(—)BB(BN),

Where:

M_{R,G,B}{R,G,B} represent the filters, which can be selected likelinear filters with a nil sum, like for example, the high-pass frequencyfilters. c_{R,G,B}{R,G,B} represents the coefficients balancing theimpact of each filter M_{R,G,B}{R,G,B}.

Such filtering example may also reflect the sharpness of the sharpestcolour onto the others. For example, supposing that only the colour blueis sharp, the high-pass filters M_{R,G,B}{R,G,B} will provide valuesnearing 0 when applied to the colours green and red which are blurred inthe example. In this particular case, GA shall thus be equal to GN plusc_GB*M_GB(BN), i.e. GN plus the high frequencies of the blue colour. Thegreen colour thus inherits the sharpness of the sharp colour (the blue).The same applies for the red colour.

In practice, the sharpness of the colours is not a binary factor; hence,the filters M_{R,G,B}{R,G,B} and the coefficients c_{R,G,B}{R,G,B} maybe adapted to the various possible values of the colour sharpness.

An embodiment example of such adaptation, in the context of RGB imagesstemming from a given capturing apparatus, is as follows:

The relative sharpness values of the red are considered in relation tothe green; likewise the blue in relation to the green: V_GR, V_GB. Suchvalues are quantified so that the values thus quantified constitute anentry into a table 2D of reasonable size. For each entry (V_BR, V_BGquantified value couples), a set of filters M_{R,G,B}{R,G,B} isassociated, as well as a set of adapted coefficients c_{R,G,B}{R,G,B}.In the particular case where we are seeking to improve the sharpness ofthe digital image, the filters M_{R,G,B}{R,G,B} and a set ofcoefficients c_{R,G,B}{R,G,B} can be pre-determined for each entry, thusensuring that the sharpness of a digital image, taken by the capturingapparatus and having relative sharpness corresponding to the entry, isperfectly corrected through application of the filter M.

It will also be possible to refine the filter M in order to take accountof the fact that the three colours do not vary on a large scale in thesame manner, notably in the coloured zones (as oppose to the black, greyor white zones) of the digital image. For that, it will be possible, forexample, to balance out, in each pixel P of the zone Z′, the actions ofthe filters M_{R,G,B}{R,G,B} by increasing the relative variations ofthe colours within a vicinity of the pixel P.

In certain case, the association table between considered relativesharpness and a set of filters may comprise other entries like, forexample, the position of the zone Z′ within the field of the image orthe parameters of the shots as the value of the focal length, of theaperture, of the focussing distance, etc., and of the optics systemduring picture-taking. Indeed, it is common that the sharpnessspecifications of a digital image also depend upon these factors.

Thereby, in order to correct the sharpness of a digital image, the imagefield will first be divided up into several zones Z′ and the method willbe applied to each one of the zones. The division shall preferably beperformed according to the sharpness specifications of the colours sothat the sharpness of the colours in each zone reveal a certain harmony.

With this embodiment can be obtained an automatic adaptation of thesharpness filtering applied to the digital image, and to the distancebetween the imaged scene and the capturing apparatus. It is also to benoted that, through using the relative sharpness, such automaticadaptation to the distance can be done without having explicit knowledgeof such distance.

Beyond the sharpness alteration of the digital image, such embodiment ofthe method also enables the automatic adaptation of processes aiming,for example, to correct optical and/or sensor defects whose effects onthe image depend on the distance between the imaged scene and thecapturing apparatus. The blur (or loss of sharpness) is an example,though other optical and/or sensor defects, such as geometricdistortions or vignetting, constitute other examples.

PRINCIPLES OF THE INVENTION Description of FIGS. 19.1, 19.2, 19.3

FIGS. 19.1, 19.2 and 19.3 show the steps of the method in accordancewith the invention, according to several embodiment modes.

FIG. 19.1 represents an image 10 comprising a region R and having twocolours 195 and 196, a measurement of relative sharpness 190 between thetwo colours 195 and 196 within the region R of the image 10, a function191 activated depending on the measured relative sharpness. As anoption, the function activated depends upon a mode 193 corresponding,for example, to a selection made by the user of the apparatus, and/or aspecification of the capturing apparatus during picture-taking.

FIG. 19.2 represents an image 10 comprising a region R and having twocolours 195 and 196, a measurement of relative sharpness 190 between thetwo colours 195 and 196 within the region R of the image 10, a function191 activated depending on the measured relative sharpness comprising aprocessing of the image 10 and producing a processed image 192. As anoption, the function activated also depends upon a mode 193corresponding, for example, to a selection made by the user of theapparatus, and/or a specification of the capturing apparatus duringpicture-taking.

FIG. 19.3 represents an image 10 comprising a region R and having twocolours 195 and 196, a measurement of relative sharpness 190 between thetwo colours 195 and 196 within the region R of the image 10, a function191 activated depending on the measured relative sharpness comprising aprocessing of another image 194 and producing a processed image 198. Asan option, the function activated also depends upon a mode 193corresponding, for example, to a selection made by the user of theapparatus, and/or a specification of the capturing apparatus duringpicture-taking.

Application on the Alteration of the Contrast and/or of the Brightnessand/or of the Colour and/or of the Sharpness

We are now going to describe an embodiment of the invention in which thefunction activation consists of modifying the image contrast and/orluminosity and/or colour, depending on the relative sharpness between atleast two colours within at least one region R of the image.

Using the relative sharpness between at least two colours in at leastone region R of the image, directly or indirectly (for example, with astep for estimating the geometry of the scene in 3 dimensions), enables,for example, to simulate the addition of a local lighting, for example,a flashlight positioned anywhere in the scene, and/or, conversely, toreduce the effect of a flash or lighting of various colours within thescene. Hence, it is possible to reduce the backlighting and flat-tinteffects of light linked to the flash.

In an embodiment, a digital image is divided into regions depending onthe relative sharpness between at least two colours, in order that eachimage region in a part of the scene is located within a range of givendistances and which is oriented in a given direction. An indication ofthe direction can be obtained using the local variation of the relativesharpness in the image. An indication of the distance can be obtainedusing the relative sharpness, as described above. It is also possible todirectly use the relative sharpness and its variation without passing bya distance and an orientation.

In order to add or modify a lighting, it is possible to determine, foreach region, the quantity and the colour of the light to be added orremoved for each point, since the distance to the simulated source inrelation to the imaged point is known, as is the orientation of theimaged object in relation to the source.

In an embodiment, the geometry of the scene in three dimensions isreconstituted by measuring the distance via a large number of points inthe image. We will thus use a known art in the field of image synthesisfor the purpose of adding lighting to the scene (ray casting or other).

In an embodiment, a lighting is added to the main subject or subjects,as adapted to each subject in order to provoke a “fill-in” effectsimulating one or several flashlights positioned opposite or on the sideof each subject. This operation can be conducted automatically andindependently for each subject. Using the known art, the addition oflighting for each subject is only possible by way of studio lighting.

Similarly, it is possible to determine the strength of the flashlightaccording to the nearest subject for the purpose of illuminating itcorrectly, and then to complete lighting on the other subjects by addinga simulated lighting.

It is also possible to determine the colour of the lighting for eachregion by way of a known method for estimating the white balance andthen to render uniform the colour of the scene's lighting. In the stateof the art, white balance is generally estimated due to lack ofinformation on the 3-dimensional geometry of the scene.

Description of FIGS. 18.1 and 18.2

FIG. 18.1 represents a sensor 2, producing a raw image 180 undergoing apre-treatment, for example white balancing, and/or a compensation of theblack level, and/or a noise reduction, in order to produce apre-processed image 182. Also represented is a relative sharpnessmeasurement 190 activating a function 191 corresponding to a processimplementing the pre-treated image 182 and the measurement of relativesharpness 190, in order to produce a processed image 192. Finally isrepresented a downstream process of the processed image 192,corresponding, for example, to a demosaicing or to other necessaryprocesses for converting a visible raw image.

FIG. 18.2 represents a sensor 2, producing a raw image 180. Alsorepresented is a relative sharpness measurement 190 activating afunction 191 corresponding to a process implementing the raw image 180and the measurement of relative sharpness 190, in order to produce aprocessed image 192. Finally is represented a downstream process of theprocessed image 192, corresponding, for example, to a demosaicing or toother necessary processes for converting a raw image into a visibleimage.

In a variant, the function implements a process on a visible image.

Simplification of the Optics

The invention is applied to an apparatus comprising variable parameterswhen capturing the digital image and having an influence on thesharpness of the colours, notably a capturing apparatus with a zoom,and/or a lens with variable focussing and/or a variable aperture. Thus,the sharpness curves 8.2 and 8.3 shall be used, corresponding to thevalue of the variable parameters according to the digital image.

As described, the invention enables to digitally restore focussingwithout a mobile unit and without delay, which thus enables to reducethe complexity of a zoom by removing at least one mobile part. Forexample, according to the distance of the subject and of the focallength, the relative sharpness between two colours can be variable,whereas this is not acceptable in the known optics.

1. A method for activating at least one function by using a measurementtaken from at least one digital image having at least two colours andoriginating from an image-capturing apparatus, wherein: a relativesharpness is measured between at least two colours on at least oneregion of the digital image, and at least one function, from a pluralityof functions, is activated depending on the measured relative sharpness,the plurality of functions including a function to determine a distancebetween the capturing apparatus and at least one object in the digitalimage based on the relative sharpness.
 2. A method according to claim 1wherein the at least one function activated is included in a groupcomprising: a determination of a relative distance between two imagedobjects, an action depending on the said distance and/or the saidrelative distance, a process on at least one zone Z′ of the digitalimage and/or of another digital image, a servo-control of the capturingapparatus and/or a servo-control of another apparatus, a provision of anindication and/or alarm and/or alert signal to a user, a detection of apart of the image, an alteration of a colour sharpness, a determinationof a position and/or of the movement of the capturing apparatus, adetermination of a subject's position within the image, an alteration ofat least one image specification, an alteration of all or part of theimage, a determination of a zone of interest inside the image, notablyin order to provide a servo-control signal, an alteration of aresolution for all or part of the image, a provision of data relating tothe image, a provision of data to a sound-capturing device, aparametering of a compression, at least one adjustment for the capturingapparatus.
 3. The method according to claim 1 or 2, wherein the functionactivated includes a process on at least one zone Z′ of the digitalimage and/or of another digital image, and the zone Z′ is included inall or part of the digital image region, and/or the whole digital image,and/or a separate zone from the digital image region, and/or anotherdigital image, and/or another whole digital image.
 4. The methodaccording to claim 3 wherein the zone Z′, for which a process isactivated, comprises at least one pixel of an image and wherein a regionR comprises a pre-determined vicinity of the corresponding pixel in thedigital image.
 5. The method according to claim 4, wherein the processis applied to all the pixels of an image.
 6. The method according toclaim 3, wherein the process on at least the zone Z′ includes alterationof at least one specification contained in the group comprising:sharpness, contrast, luminosity, detail, colour, the type ofcompression, the rate of compression, the image contents, theresolution.
 7. The method according to claim 3, wherein the processincludes an alteration of the sharpness of each pixel in the zone Z′ byway of a filter mixing the values attached to the pixel or pixels withina pre-determined vicinity for each pixel, the parameters of the filterdepending upon the measured relative sharpness.
 8. The method accordingto claim 3, wherein the zone Z′ is determined by using the measuredrelative sharpness.
 9. The method according to claim 3, wherein the zoneZ′ is included in a background for an image, notably destined for remotetransmission, through a system of visio or video-conferencing.
 10. Themethod according to claim 3, wherein the process includes the provisionof data depending on the distance between the imaged object and thecapturing apparatus for all or part of the pixels of the zone Z′, andthe storage and/or the transmission and/or the use of such data isactivated depending on the distance, the stored data notably being savedin a data-processing file, in an image file.
 11. The method according toclaim 1, wherein the function activated includes a servo-control of thecapturing apparatus, such as a servo-control for focussing and/or ofexposure, and/or of flash, and/or for image-framing, and/or forwhite-balancing, and/or for image-stabilising, and/or a servo-control ofanother apparatus or device linked to the capturing apparatus, includingguiding of a robot.
 12. The method according to claim 1, wherein thefunction activated includes a provision of a signal, including anindication signal of the main focal point of the digital image and/or ofa focussing zone and/or of the distance of at least one part of theimaged scene, to the capturing apparatus.
 13. The method according toclaim 1, wherein the function activated depends upon at least onespecification of the capturing apparatus during picture-taking includingthe focal length, the aperture, the distance for focussing, the exposureparameters, the white-balance parameters, the resolution, thecompression, or an adjustment made by the user.
 14. The method accordingto claim 1, wherein the digital image is included in a raw imagestemming from the sensor of the capturing apparatus.
 15. The methodaccording to claim 1, wherein the measurement is made inside thecapturing apparatus and/or the function activated is made inside thecapturing apparatus.
 16. The method according to claim 1, wherein themeasurement is taken beyond the capturing apparatus, including on acomputer after transfer of the image, and/or wherein a function isactivated beyond the capturing apparatus.
 17. The method according toclaim 1, wherein the function activated comprises a detection and/orrecognition activation of a part of the image, including detectionand/or recognition of the face.
 18. The method according to claim 1,wherein the measurement comprises a measurement of relative sharpnessbetween a first colour and at least a second colour, referred to as theother colour, and wherein the activation comprises alteration of thesharpness of the other colour depending on the sharpness of the firstcolour.
 19. The method according to claim 18 wherein the sharpnessalteration of the other colour is embodied using a calculation of thetype CA=CN+F(CO−CN), where CA represents the other improved colour, COrepresents the other colour prior to processing, CN represents the firstcolour and F represents a filter.
 20. The method according to claim 18,wherein alteration of the sharpness of the other colour is animprovement of the sharpness, the first colour being referred to as the“sharp colour”.
 21. The method according to claim 20 further comprising:decomposing the digital image into regions, the sharp colour beingselected one for each region.
 22. The method according to claim 20, thesaid capturing apparatus comprising a macro mode, a method wherein thesharp colour is selected depending on activation of the macro mode. 23.The method according to claim 18, wherein the sharpness of at least onecolour within at least one image zone is reduced.
 24. The methodaccording to claim 1, for which the said digital image stems from acapturing apparatus comprising a lens, the said method further includingthe step to select a lens from among a series of pre-determined lenses,the said lens representing specifications such that the images of anobject with at least two distinct pre-determined distances representdistinct colours of sharpness, thus improving the depth of field andreducing the cost of the lens.
 25. The method according to claim 1,wherein the function activated comprises a measurement of the positionand/or movement of the capturing apparatus.
 26. The method according toclaim 1, wherein the function activated involves determination of theposition of the main subject or subjects in the image.
 27. The methodaccording to claim 26 wherein the function activated further involvesthe automatic framing, including the centering, zooming or reframing ofthe digital image, and/or of another image, in relation to the mainsubject of the digital image.
 28. The method according to claim 1,wherein the function activated comprises the application of a processdepending on the relative sharpness and on the user's choice.
 29. Themethod according to claim 1, wherein the function activated comprisesalteration of the contrast and/or of the brightness and/or of the colourand/or of the sharpness of an image, depending on the variation of therelative sharpness within the image.
 30. The method according to claim1, wherein the function activated comprises provision of the position ofat least one zone of interest for consideration to a servo-control ofexposure and/or of white balance and/or of focussing, such zone ofinterest being determined by comparing at least two relative sharpnessmeasurements.
 31. The method according to claim 1, wherein the functionactivated comprises provision of a signal to the user, indicating thatthe image is taken too close-up to be sharp.
 32. The method according toclaim 1, wherein the function activated comprises alteration of an imageresolution depending on the relative sharpness.
 33. The method accordingto claim 1, wherein the function activated comprises the provision of asignal used for an automatic indexing of the digital image.
 34. Themethod according to claim 1, wherein the function activated comprisesprovision of remote or directional data, in relation to the capturingapparatus, of a subject or an object of the digital image, to asound-capturing device.
 35. The method according to claim 1, wherein thefunction activated includes the parametering of increased compressionfor the background, and of reduced compression for the main subject orsubjects, such main subject or subjects being determined as an imagezone complying with the criteria based upon the measured relativesharpness.
 36. The method according to claim 1, wherein the apparatuscomprises a sensor having pixels equipped with coloured filters of atleast two types, the said filters being selected so that their spectralresponses entail little overlapping.
 37. The method according to claim1, wherein the apparatus comprises a sensor having pixels mainly servingto generate the image, and other pixels mainly serving to measure therelative sharpness.
 38. The method according to claim 37, wherein thepixels of the sensor, mainly serving to measure the relative sharpness,have a spectral response within a spectral band which entails littleoverlapping with the spectral band of the sensor's pixels, mainlyserving to generate the image.
 39. The method according to claim 37wherein the pixels of the sensor, mainly serving to generate an image,have a spectral response that is mainly within the field visible to thehuman eye, and wherein the other pixels have a spectral response, mainlybeyond the field visible to the human eye.
 40. The method according toclaim 1, in which the capturing apparatus includes a sensor comprisingspectral-response pixels, mainly within the field visible to the humaneye, and additional pixels having a spectral response, mainly beyond thespectre visible to the human eye, and where the sharpness of the part ofthe image stemming from these additional pixels exceeds, within at leastone range of distances between the capturing apparatus and the imagedscene, the sharpness of the part of the image supplied by the responsepixels, mainly within the field visible to the human eye.
 41. The methodaccording to claim 1, wherein the apparatus is equipped with a lenslacking a mobile or flexible element for focussing.
 42. The methodaccording to claim 1, wherein the apparatus is equipped with a lenshaving a variable focal length with no mobile or flexible element forfocussing.
 43. The method according to claim 1, wherein the apparatus isequipped with a lens having a variable focal length with a single mobileor flexible optics group.
 44. The method according to claim 1, whereinthe digital image stems from at least two sensors.
 45. The methodaccording to claim 1, wherein the function activated comprises theaddition of an object inside an image and/or the replacement of a partof an image depending on the measured relative sharpness on the digitalimage.
 46. The method according to claim 1, wherein the capturingapparatus capturing a sequence of images, the digital image being a partof the sequence, the action is performed on at least one other image ofthe sequence.
 47. The method according to claim 1, wherein the functionactivated alters at least one adjustment of the capturing apparatus,including the focal length, the aperture, or the distance for focussing.48. The method according to claim 1, wherein the function activatedcomprises the production of an altered raw image.
 49. The methodaccording to claim 1, wherein the apparatus comprising a lens havinglongitudinal chromatic aberrations, including for a given focus,aperture and focal distance, there exists at least one colour for whichthe distance involving the best sharpness is lower than${k\;\frac{f^{2}}{O \cdot P}},$ k being a coefficient lower than 0.7, fbeing the focal length, O being the aperture and P having the smallest,from among all the colours of the image, diameter of the blur spot of anobject point situated in infinity.
 50. The method according to claim 1,wherein the measurement of relative sharpness between two colours ismade by comparing the results of a first measurement applied to thefirst colour and the results of a second measurement applied to a secondcolour, each measurement being such that it depends upon, on the onehand, the colour, and upon the contents of the digital image on theother, the comparison being such that it is removed from the digitalimage contents.
 51. The method according to claim 1, the said digitalimage stemming from a capturing apparatus comprising a lens, the saidmethod further comprising the step to design a lens representingspecifications such that the images of an object with at least twopre-determined distances represent distinct sharpness colours: in orderthat the depth of field and/or the aperture and/or every other opticalspecification is improved, and/or that the cost of the optics isreduced.
 52. An image-capturing and/or reproducing apparatus comprisingan optical system for capturing and/or reproducing images, an imagesensor and/or generator and/or a servo-control system, the image beingprocessed, in view of its improvement, by digital image-processingmeans, the method being such that the user determines or selects theoptical system and/or the sensor and/or the image generator and/or theservo-control system parameters, using the capacity to process imagesvia digital means, and further determines or selects improvement of acolour sharpness depending on another colour in accordance with a methodaccording to claim 1, in order to minimise the embodiment costs and/orto optimise the performances of the image-capturing and/or reproducingapparatus.
 53. An image-capturing apparatus using the method accordingto claim
 1. 54. A digital image-processing device using the methodaccording to claim
 1. 55. A method according to claim 1 enabling toimprove sharpness of at least one colour of a digital image, notablystemming from an image-capturing device, comprising: selecting fromamong a plurality of colours at least one colour referred to as “sharpcolour;” and reflecting the sharpness of the sharp colour onto at leastone other improved colour, so that the improved colour representsincreased sharpness.
 56. The method according to claim 55 furthercomprising: decomposing the digital image into regions, the said sharpcolour being selected for each region.
 57. The method according to claim56, the said digital image stemming from a capturing apparatus, the saidmethod further comprising: determining the distance between thecapturing apparatus and at least one object of the captured scene byusing the sharpness of at least two colours in an image region of thesaid object.
 58. The method according to claim 57 further comprising:reducing the sharpness of at least one colour within at least one imageregion.
 59. The method according to claim 57 further comprising:determining a servo-control instruction for the said capturing apparatusby using the sharpness of at least two colours in order that focussingis achieved in fewer steps and is accelerated.
 60. The method accordingto claim 55, the said sharp colour selection being that of choosing thesharpest colour according to a pre-determined rule.
 61. The methodaccording to claim 55, the said “sharp colour” selection beingpre-determined.
 62. The method according to claim 55, the said digitalimage stemming from a capturing apparatus, the said sharp colourselection depending on the distance between the capturing apparatus andat least one object of the captured scene in order to obtain the saiddigital image.
 63. The method according to claim 55, the said imageapparatus comprising a macro mode, the said sharp colour selectiondepending upon activation of the macro mode.
 64. The method according toclaim 55, the said digital image stemming from a capturing apparatuscomprising a lens, the said method further comprising: selecting a lensfrom among a series of pre-determined lenses, the said lens representingspecifications such that the images of an object having at least twopre-determined distances represent distinct sharp colours, thusimproving the depth of field and/or reducing the cost of the lens. 65.The method according to claim 55, wherein repercussion of the sharpnessof the sharp colour on at least one other improved colour is embodied byusing a calculation of the type CA=CN+F(CO−CN), where CA represents theimproved colour, CO represents the improved colour prior to processing,CN represents the sharp colour and F represents a filter.
 66. The methodaccording to claim 55, the said digital image stemming from a capturingapparatus comprising a lens, the said method further comprising:designing a lens by taking account of the method according to theinvention, the said lens representing specifications such that theimages of an object with at least two pre-determined distances representdistinct sharp colours: in order that the depth of field and/or theaperture and/or every other optical specification is improved and/or thecost of the optics is reduced, and in order that the mechanicalfocussing can be achieved by using fewer positions.
 67. Animage-capturing and/or reproducing apparatus comprising an opticalsystem for capturing and/or reproducing images, an image sensor and/orgenerator and/or a servo-control system, the image being processed, inview of its improvement, by digital image-processing means, the methodbeing such that the user determines or selects the optical system and/orthe sensor and/or the image generator and/or the servo-control systemparameters, using the capacity to process images via digital means, andespecially for improving the sharpness of a colour depending on thesharpness of another colour in accordance with a method according toclaim 55, in order to minimise the embodiment costs and/or to optimisethe performances of the image-capturing and/or reproducing apparatus.68. The image-capturing and/or reproducing apparatus using thecolour-improvement method according to claim
 55. 69. A digitalimage-processing device implementing the method according to claim 55.70. A digital image obtained using the image-capturing and/or thereproducing apparatus according to claim 68.